Oct 30 21:39:47.368573 ip-10-0-134-73 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:39:47.368585 ip-10-0-134-73 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:39:47.368592 ip-10-0-134-73 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:39:47.368859 ip-10-0-134-73 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:39:57.617854 ip-10-0-134-73 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:39:57.617872 ip-10-0-134-73 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1529fa245dee4742ab764e180d3b1843 -- Oct 30 21:41:55.878576 ip-10-0-134-73 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:41:56.415750 ip-10-0-134-73 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:56.415750 ip-10-0-134-73 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:41:56.415750 ip-10-0-134-73 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:56.415750 ip-10-0-134-73 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:56.419824 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419712 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.419824 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419824 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419828 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419833 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419836 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419839 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419842 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419844 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419847 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419849 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419852 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419854 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419857 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419859 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419862 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419864 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419867 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419876 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419878 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419881 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419883 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.419894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419886 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419888 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419891 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419893 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419896 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419899 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419902 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419904 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419907 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419910 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419913 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419915 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419917 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419920 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419923 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419925 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419928 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419930 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419932 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419935 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.420368 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419937 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419939 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419942 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419944 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419946 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419949 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419951 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419954 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419956 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419959 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419962 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419965 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419967 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419970 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419972 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419975 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419977 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419987 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419991 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.420829 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419995 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.419999 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420001 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420004 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420007 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420009 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420012 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420014 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420017 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420020 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420022 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420025 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420028 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420030 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420033 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420036 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420038 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420041 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420043 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.421327 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420046 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420049 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420051 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420053 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420056 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420059 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420061 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420064 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420067 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420070 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420072 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420074 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420077 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420079 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420082 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420084 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420087 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420089 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420092 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420094 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.421776 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420097 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420099 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420104 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420107 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420110 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420113 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.420116 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421614 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421622 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421626 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421629 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421632 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421634 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421637 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421639 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421642 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421645 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421648 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421651 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.422245 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421654 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421657 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421660 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421663 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421665 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421668 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421671 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421673 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421676 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421678 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421680 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421683 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421686 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421688 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421691 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421693 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421696 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421699 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421702 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421704 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.422700 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421707 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421709 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421712 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421714 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421717 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421719 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421722 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421724 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421727 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421729 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421732 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421735 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421738 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421740 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421743 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421747 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421751 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421756 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421759 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.423177 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421761 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421764 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421766 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421769 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421771 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421774 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421776 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421778 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421781 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421785 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421788 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421790 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421793 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421796 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421798 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421801 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421803 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421806 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421808 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.423697 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421811 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421813 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421816 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421818 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421821 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421823 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421826 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421829 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421832 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421834 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421836 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421839 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421841 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421844 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421846 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421849 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421851 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421854 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421856 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421859 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.424145 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421862 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421864 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421867 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421870 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421872 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421875 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421877 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421879 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421883 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421886 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421889 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421891 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421894 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421896 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421899 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.421901 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422429 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422435 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422439 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.424711 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422442 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422444 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422447 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422450 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422453 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422455 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422458 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422460 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422463 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422466 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422469 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422471 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422474 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422477 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422479 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422482 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422484 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422487 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422490 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422492 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.425172 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422495 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422498 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422500 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422503 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422505 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422508 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422510 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422513 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422515 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422518 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422521 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422523 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422526 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422528 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422532 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422534 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422536 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422539 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422541 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422544 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.425657 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422546 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422548 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422551 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422553 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422555 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422558 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422560 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422563 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422566 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422569 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422571 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422574 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422577 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422581 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422585 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422588 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422591 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422594 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422596 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.426102 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422599 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422601 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422604 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422606 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422608 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422611 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422614 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422616 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422619 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422621 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422624 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422626 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422629 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422631 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422633 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422636 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422638 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422641 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422643 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422645 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.426572 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422648 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422651 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422655 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422659 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422662 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422666 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422669 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422671 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422673 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422676 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422678 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422681 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422684 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422686 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422688 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422691 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422693 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422695 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422698 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.427035 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422700 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422704 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422707 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422710 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.422712 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423431 2570 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423440 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423446 2570 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423451 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423456 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423459 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423464 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423467 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423471 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423474 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423478 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423481 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423484 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423487 2570 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423490 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423493 2570 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423496 2570 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423499 2570 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423504 2570 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:41:56.427541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423507 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423510 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423513 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423516 2570 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423520 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423523 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423526 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423530 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423532 2570 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423536 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423539 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423542 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423545 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423549 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423552 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423555 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423558 2570 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423562 2570 flags.go:64] FLAG: --enable-server="true" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423565 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423569 2570 flags.go:64] FLAG: --event-burst="100" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423572 2570 flags.go:64] FLAG: --event-qps="50" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423575 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423578 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423581 2570 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:41:56.428092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423585 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423588 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423594 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423597 2570 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423600 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423603 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423606 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423608 2570 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423611 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423614 2570 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423617 2570 flags.go:64] FLAG: --feature-gates="" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423621 2570 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423624 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423627 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423630 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423634 2570 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423637 2570 flags.go:64] FLAG: --help="false" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423640 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423643 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423646 2570 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423649 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423652 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423656 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:41:56.428715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423659 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423662 2570 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423664 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423667 2570 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423670 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423673 2570 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423676 2570 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423679 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423681 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423685 2570 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423687 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423691 2570 flags.go:64] FLAG: --lock-file="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423694 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423697 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423700 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423709 2570 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423711 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423715 2570 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423717 2570 flags.go:64] FLAG: --logging-format="text" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423720 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423724 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423727 2570 flags.go:64] FLAG: --manifest-url="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423729 2570 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423734 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423736 2570 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:41:56.429253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423740 2570 flags.go:64] FLAG: --max-pods="110" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423743 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423746 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423750 2570 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423752 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423756 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423758 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423761 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423769 2570 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423772 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423774 2570 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423777 2570 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423780 2570 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423783 2570 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423785 2570 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423788 2570 flags.go:64] FLAG: --port="10250" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423791 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423794 2570 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e343f3552af8dc5b" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423798 2570 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423801 2570 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423804 2570 flags.go:64] FLAG: --register-node="true" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423807 2570 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423811 2570 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423813 2570 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423816 2570 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423819 2570 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:41:56.429865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423822 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423825 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423828 2570 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423831 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423834 2570 flags.go:64] FLAG: --runonce="false" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423836 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423839 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423842 2570 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423845 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423848 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423851 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423855 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423857 2570 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423860 2570 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423863 2570 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423866 2570 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423869 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423872 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423874 2570 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423877 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423882 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423885 2570 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423887 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423892 2570 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423896 2570 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:41:56.430471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423898 2570 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423901 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423903 2570 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423908 2570 flags.go:64] FLAG: --v="2" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423913 2570 flags.go:64] FLAG: --version="false" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423917 2570 flags.go:64] FLAG: --vmodule="" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423920 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.423923 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424059 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424062 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424065 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424067 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424070 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424072 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424075 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424077 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424080 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424082 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424085 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424088 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424090 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.431410 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424095 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424099 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424102 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424105 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424108 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424110 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424113 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424115 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424118 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424120 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424125 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424127 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424130 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424132 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424136 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424139 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424141 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424144 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424147 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424150 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.431931 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424152 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424155 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424159 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424162 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424165 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424167 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424171 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424173 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424176 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424178 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424181 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424184 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424186 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424189 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424191 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424193 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424196 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424198 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424201 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.432435 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424203 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424206 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424208 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424212 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424214 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424216 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424219 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424222 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424225 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424228 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424230 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424233 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424235 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424238 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424240 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424243 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424245 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424248 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424250 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424253 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.432905 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424255 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424258 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424260 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424263 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424265 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424268 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424270 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424273 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424275 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424278 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424280 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424283 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424285 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424288 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424290 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424294 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424296 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424299 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424319 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424325 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.433386 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424329 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424331 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424334 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424336 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424339 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424341 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424344 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424346 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424349 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424352 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424354 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424357 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424359 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424362 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.424389 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424425 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.433850 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424430 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424433 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424437 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424439 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424442 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424444 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424447 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424449 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424452 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424455 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424457 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424461 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424464 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424467 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424469 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424472 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424476 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424479 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424481 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424483 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.434212 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424486 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424488 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424491 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424493 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424496 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424498 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424501 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424503 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424506 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424508 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424510 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424513 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424515 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424518 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424521 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424524 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424526 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424529 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424531 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424533 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.434704 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424536 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424538 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424541 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424544 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424547 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424549 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424552 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424554 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424558 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424560 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424564 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424569 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424571 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424574 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424577 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424580 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424583 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424587 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424589 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.435163 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424592 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424594 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424597 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424599 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424602 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424604 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424607 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424609 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424612 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424615 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424617 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424620 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424622 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424626 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424629 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424632 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424635 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424638 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424640 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.435636 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424643 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424645 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424649 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424652 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424654 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424656 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424659 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424661 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424664 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424666 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424669 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424671 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424674 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424676 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424678 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424681 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424683 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424685 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424689 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424691 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.436079 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424694 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424696 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424699 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424702 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424704 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424707 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.424709 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.432099 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.432226 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.432249 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.432255 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432335 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432341 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432344 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432347 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432350 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432355 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432359 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432362 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.436553 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432365 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432368 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432370 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432373 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432376 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432378 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432381 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432384 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432387 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432390 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432393 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432396 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432399 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432401 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432404 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432406 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432409 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432411 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432414 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432416 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.437002 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432419 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432422 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432425 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432428 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432430 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432433 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432435 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432438 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432440 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432443 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432446 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432449 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432452 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432454 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432457 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432460 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432462 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432465 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432468 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.437479 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432471 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432473 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432476 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432478 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432481 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432483 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432486 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432488 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432491 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432493 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432495 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432498 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432500 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432503 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432506 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432508 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432511 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432514 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432516 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432518 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.437911 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432521 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432523 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432526 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432529 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432531 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432534 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432536 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432538 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432541 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432543 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432545 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432548 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432550 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432553 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432556 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432558 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432560 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432563 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432566 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432568 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.438515 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432571 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432573 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432575 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432578 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432580 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432582 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432585 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432588 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432592 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432595 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432598 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432600 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432603 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432606 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432608 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432610 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432613 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432615 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.438974 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432618 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.432644 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432790 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432794 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432797 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432800 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432803 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432806 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432809 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432811 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432814 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432817 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432819 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432822 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432824 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432827 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.439406 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432830 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432832 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432834 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432837 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432840 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432843 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432845 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432849 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432851 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432853 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432856 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432858 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432861 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432863 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432866 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432868 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432870 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432873 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432875 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432878 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.439831 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432880 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432882 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432885 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432887 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432891 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432893 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432896 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432898 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432900 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432903 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432905 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432908 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432910 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432913 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432916 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432919 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432921 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432924 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432926 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432928 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.440329 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432931 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432933 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432935 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432938 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432940 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432942 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432945 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432947 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432949 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432952 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432954 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432957 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432959 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432963 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432966 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432970 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432973 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432976 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432979 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.440894 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432981 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432984 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432986 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432989 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432991 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432993 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432995 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.432998 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433001 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433003 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433005 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433008 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433010 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433013 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433015 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433017 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433020 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433022 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433024 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433027 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.441441 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433029 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433032 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433034 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433037 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433039 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433042 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433044 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433047 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433049 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433053 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433057 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433060 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433063 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.433086 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433119 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433123 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:56.442069 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433126 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433129 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433131 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433134 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433137 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433139 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433141 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433145 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433149 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433151 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433154 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433157 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433159 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433162 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433164 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433167 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433169 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433172 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433174 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:56.442472 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433176 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433179 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433181 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433184 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433186 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433188 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433191 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433193 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433195 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433198 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433201 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433203 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433206 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433208 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433210 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433213 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433215 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433218 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433220 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433223 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:56.443178 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433225 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433228 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433230 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433234 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433237 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433240 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433242 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433245 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433247 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433250 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433252 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433254 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433257 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433259 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433262 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433264 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433267 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433270 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433272 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:56.443743 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433275 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433277 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433279 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433282 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433285 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433287 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433290 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433292 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433296 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433298 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433318 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433320 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433323 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433325 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433328 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433330 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433332 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433335 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433337 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433340 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:56.444168 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433342 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433344 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433347 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433349 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433358 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433360 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433363 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433366 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433369 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433371 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433374 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433376 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433378 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433381 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433383 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433387 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433390 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433393 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433395 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433398 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:56.444813 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433401 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433403 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433406 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433408 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433411 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:56.433413 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.433467 2570 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.435292 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.436842 2570 server.go:1014] "Starting client certificate rotation" Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.436939 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.436975 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.438570 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:41:56.445297 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.439625 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:41:56.448583 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.448569 2570 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:41:56.450700 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.450681 2570 log.go:25] "Validated CRI v1 image API" Oct 30 21:41:56.451571 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.451551 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:41:56.454737 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.454713 2570 fs.go:135] Filesystem UUIDs: map[159e24d9-df56-442d-bd8a-07b8d8815401:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dfb2a9ed-e6a8-49e9-8fd8-5348d3940aae:/dev/nvme0n1p4] Oct 30 21:41:56.454789 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.454738 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:41:56.460008 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.459889 2570 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:56.458735112 +0000 UTC m=+0.297616816 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3175357 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec243e6063db9c2b43a85d9deacaaa69 SystemUUID:ec243e60-63db-9c2b-43a8-5d9deacaaa69 BootID:1529fa24-5dee-4742-ab76-4e180d3b1843 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:c7:2e:de:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:c7:2e:de:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:2b:8c:27:06:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:41:56.460008 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.460007 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:41:56.460123 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.460111 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:41:56.461215 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.461194 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:41:56.461401 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.461223 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-73.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:41:56.461442 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.461403 2570 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:41:56.461442 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.461411 2570 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:41:56.461442 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.461438 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:41:56.461521 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.461449 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:41:56.462078 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.462069 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:41:56.464501 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.464490 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:41:56.464779 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.464769 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:41:56.465398 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.465388 2570 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:41:56.465450 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.465409 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:41:56.465450 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.465422 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:41:56.465450 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.465431 2570 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:41:56.465450 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.465438 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:41:56.465931 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.465915 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:41:56.469141 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.469109 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:41:56.473138 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.473123 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:41:56.474070 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474057 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:41:56.474114 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474088 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:41:56.474240 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474230 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:41:56.474277 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474246 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:41:56.474277 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474252 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:41:56.474277 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474257 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:41:56.474277 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474263 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:41:56.474277 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.474268 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:41:56.475343 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475334 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:41:56.475381 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475344 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:41:56.475381 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475351 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:41:56.475381 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475357 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:41:56.475381 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475367 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:41:56.475381 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475375 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:41:56.475918 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475908 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:41:56.475918 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.475918 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:41:56.479562 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.479549 2570 server.go:1263] "Started kubelet" Oct 30 21:41:56.479726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.479687 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:41:56.479885 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.479669 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:41:56.479932 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.479912 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:41:56.480233 ip-10-0-134-73 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:41:56.482690 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.480748 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:41:56.482690 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.481451 2570 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485074 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485090 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.485208 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485584 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485589 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485607 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:41:56.485743 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.485689 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:41:56.486064 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485813 2570 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:41:56.486064 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.485822 2570 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:41:56.488131 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.486118 2570 factory.go:55] Registering systemd factory Oct 30 21:41:56.488131 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.486982 2570 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:41:56.488458 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.488190 2570 factory.go:153] Registering CRI-O factory Oct 30 21:41:56.488458 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.488203 2570 factory.go:223] Registration of the crio container factory successfully Oct 30 21:41:56.488458 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.488243 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:41:56.488458 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.488260 2570 factory.go:103] Registering Raw factory Oct 30 21:41:56.488458 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.488274 2570 manager.go:1199] Started watching for new ooms in manager Oct 30 21:41:56.489339 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.489250 2570 manager.go:319] Starting recovery of all containers Oct 30 21:41:56.499501 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.499399 2570 manager.go:324] Recovery completed Oct 30 21:41:56.505427 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.505410 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.508245 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.508226 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.508396 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.508257 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.508396 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.508267 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.509470 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.509457 2570 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:41:56.509527 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.509470 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:41:56.509527 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.509490 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:41:56.509527 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.509500 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.510509 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.510494 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.510587 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.510520 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.510587 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.510533 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.510698 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.510642 2570 policy_none.go:50] "Start" Oct 30 21:41:56.510698 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.510655 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:41:56.510698 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.510670 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:41:56.511808 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.511794 2570 policy_none.go:44] "Start" Oct 30 21:41:56.538867 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.538840 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:41:56.541160 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541149 2570 manager.go:341] "Starting Device Plugin manager" Oct 30 21:41:56.541218 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.541174 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:41:56.541218 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541183 2570 server.go:87] "Starting device plugin registration server" Oct 30 21:41:56.541455 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541443 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:41:56.541487 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541468 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:41:56.541598 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541578 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:41:56.541678 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541667 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:41:56.541678 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.541677 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:41:56.542136 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.542114 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:41:56.542206 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.542179 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" Oct 30 21:41:56.618820 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.618794 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:41:56.620651 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.620046 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:41:56.620651 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.620074 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:41:56.620651 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.620102 2570 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:41:56.620651 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.620144 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:41:56.625488 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.625464 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-73.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:41:56.625488 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.625471 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-73.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:41:56.626350 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.625462 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-73.us-west-1.compute.internal.18e36ca1422f55dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-73.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-73.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:56.479522269 +0000 UTC m=+0.318403971,LastTimestamp:2026-10-30 21:41:56.479522269 +0000 UTC m=+0.318403971,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-73.us-west-1.compute.internal,}" Oct 30 21:41:56.641940 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.641920 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.642706 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.642680 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.642786 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.642721 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.642786 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.642736 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.642786 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.642759 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.713898 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.713812 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:41:56.713898 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.713857 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n5ksg" Oct 30 21:41:56.720853 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.720835 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal"] Oct 30 21:41:56.720911 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.720888 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.721784 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.721766 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.721863 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.721796 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.721863 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.721806 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.723906 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.723894 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.724064 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724048 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.724126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724087 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.724651 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724636 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.724732 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724657 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.724732 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724667 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.724732 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724720 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.724839 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724740 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.724839 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.724749 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.726707 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.726692 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.726770 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.726722 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:56.727518 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.727499 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:56.727577 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.727536 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:56.727577 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.727551 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:56.762771 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.762752 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.765904 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.765889 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.787819 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.787799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.787872 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.787823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.787872 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.787841 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0994e35dd707326725d576ce662475d4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"0994e35dd707326725d576ce662475d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.871185 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.871165 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:56.878411 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.878392 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:56.878871 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.878857 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:56.878913 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.878905 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:56.879885 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.879871 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.879938 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:56.879892 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-73.us-west-1.compute.internal\": node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" Oct 30 21:41:56.888517 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.888502 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.888573 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.888522 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.888573 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.888540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0994e35dd707326725d576ce662475d4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"0994e35dd707326725d576ce662475d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.888573 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.888564 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0994e35dd707326725d576ce662475d4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"0994e35dd707326725d576ce662475d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.888677 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.888580 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.888677 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.888593 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:56.985965 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:56.985907 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:57.046604 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.046587 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:41:57.047267 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.047256 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:57.065733 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.065706 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:57.068619 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.068603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:41:57.110927 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.110906 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:41:57.436168 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.436090 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:41:57.436713 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.436264 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:41:57.436713 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.436279 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:41:57.436713 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.436327 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:41:57.436930 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.436330 2570 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad8669e46bc8f4e2bbe5ac45b704f8a7-5cb0796d801b0643.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-73.us-west-1.compute.internal.18e36ca143e6099b\": read tcp 10.0.134.73:44054->13.184.210.151:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-73.us-west-1.compute.internal.18e36ca143e6099b default 5191 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-73.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-73.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-73.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:56 +0000 UTC,LastTimestamp:2026-10-30 21:41:56.642739776 +0000 UTC m=+0.481621477,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-73.us-west-1.compute.internal,}" Oct 30 21:41:57.465642 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.465620 2570 apiserver.go:52] "Watching apiserver" Oct 30 21:41:57.486151 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.486122 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:41:57.718538 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.718453 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:57 +0000 UTC" deadline="2028-06-30 03:06:20.370884082 +0000 UTC" Oct 30 21:41:57.718538 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.718492 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14597h24m22.652394991s" Oct 30 21:41:57.738541 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.738518 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:41:57.738812 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.738796 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-qg64f","kube-system/global-pull-secret-syncer-56vhn","kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-ng9zm","openshift-ovn-kubernetes/ovnkube-node-7mj55","kube-system/konnectivity-agent-cggst","openshift-cluster-node-tuning-operator/tuned-xfvkg","openshift-network-operator/iptables-alerter-5kndd","openshift-multus/multus-additional-cni-plugins-8hwrm","openshift-multus/network-metrics-daemon-bpfn7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct","openshift-dns/node-resolver-tsn5z","openshift-image-registry/node-ca-2r78l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal"] Oct 30 21:41:57.741064 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.741044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.743001 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.742981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.743087 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.743036 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:41:57.744883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.744870 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:41:57.744936 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.744921 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:41:57.746938 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.746927 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.748933 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.748913 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.751059 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.751046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.753147 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.753129 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.755351 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.755335 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.757447 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.757434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:57.757507 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.757479 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:41:57.759677 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.759661 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.761921 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.761905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.763936 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.763920 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.787067 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.787052 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:41:57.793551 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793530 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-var-lib-kubelet\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.793684 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cnibin\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.793684 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:57.793684 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcp62\" (UniqueName: \"kubernetes.io/projected/9bc0c153-0125-47d4-bf2d-27668920eb0b-kube-api-access-tcp62\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.793684 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-kubernetes\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-run\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793745 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793745 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793764 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-etc-tuned\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793820 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793851 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qtzdm\"" Oct 30 21:41:57.793883 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793870 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcgb\" (UniqueName: \"kubernetes.io/projected/d86f6ae9-bad4-41c8-be3c-61e4225a0501-kube-api-access-qvcgb\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793914 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793956 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-sys\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.793983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-sys-fs\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794006 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-config\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794055 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft54x\" (UniqueName: \"kubernetes.io/projected/eb2cef57-0b45-422b-9390-ee103989900e-kube-api-access-ft54x\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7xz\" (UniqueName: \"kubernetes.io/projected/41738995-836d-4314-baa7-d2e7f5149a56-kube-api-access-rp7xz\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvv6\" (UniqueName: \"kubernetes.io/projected/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-kube-api-access-pwvv6\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-etc-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.794204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d86f6ae9-bad4-41c8-be3c-61e4225a0501-host\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794220 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-multus\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-daemon-config\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-netd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb80ad79-d9ca-41d8-a676-8844585591d4-iptables-alerter-script\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2cef57-0b45-422b-9390-ee103989900e-tmp-dir\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-systemd\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794440 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794518 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-device-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-node-log\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794581 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-agent-certs\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.794672 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794628 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-hostroot\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-k8s-cni-cncf-io\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-multus-certs\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794803 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovn-node-metrics-cert\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794825 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-conf-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-etc-kubernetes\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-dbus\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794908 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-kubelet\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.794990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-bin\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-system-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-socket-dir-parent\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.795170 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-kubelet-config\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.795691 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-tmp\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.795691 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.795691 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-socket-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.795691 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-slash\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.795691 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-systemd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.795691 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795658 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2cef57-0b45-422b-9390-ee103989900e-hosts-file\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795711 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-kubelet\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-cnibin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysconfig\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb80ad79-d9ca-41d8-a676-8844585591d4-host-slash\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795830 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-cni-binary-copy\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795850 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-systemd-units\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zqr\" (UniqueName: \"kubernetes.io/projected/eb80ad79-d9ca-41d8-a676-8844585591d4-kube-api-access-s4zqr\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-conf\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795913 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-bin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-netns\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.796034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.795985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8vd\" (UniqueName: \"kubernetes.io/projected/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kube-api-access-fm8vd\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-var-lib-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-system-cni-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-registration-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-ovn\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796453 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnhk\" (UniqueName: \"kubernetes.io/projected/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-kube-api-access-vsnhk\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-host\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796534 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-netns\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796602 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796582 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796621 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d86f6ae9-bad4-41c8-be3c-61e4225a0501-serviceca\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-lib-modules\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f749v\" (UniqueName: \"kubernetes.io/projected/6b6edcdf-aa04-4afd-870c-4670f562dc06-kube-api-access-f749v\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796730 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-log-socket\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796813 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-env-overrides\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-script-lib\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796878 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-konnectivity-ca\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:41:57.796987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.796970 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-modprobe-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.798786 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.798754 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:41:57.798902 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.798860 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.798978 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.798919 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:41:57.798978 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.798934 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.799071 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.799048 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:41:57.799269 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.799250 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:41:57.799785 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.799770 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pvwl\"" Oct 30 21:41:57.800529 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.800512 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:41:57.800641 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.800617 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:41:57.800696 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.800685 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4c9l\"" Oct 30 21:41:57.802346 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.802330 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.802404 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.802333 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.802521 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.802509 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpjmc\"" Oct 30 21:41:57.839111 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.839093 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.839253 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.839241 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.839331 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.839257 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vmmq\"" Oct 30 21:41:57.839331 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.839263 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:41:57.846647 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.846630 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qdnct\"" Oct 30 21:41:57.846765 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.846750 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fxktb\"" Oct 30 21:41:57.846878 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.846858 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:41:57.847074 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.847062 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xw9dh\"" Oct 30 21:41:57.850713 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.850700 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rfsrz\"" Oct 30 21:41:57.851227 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.851198 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:41:57.851288 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.851250 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k4spp" Oct 30 21:41:57.851915 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.851900 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.852044 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852029 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.852126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852062 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:41:57.852126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852075 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.852187 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852141 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:41:57.852187 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852160 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.852187 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852161 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:41:57.852335 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852293 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:41:57.852372 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.852354 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:41:57.897687 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-env-overrides\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-script-lib\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-konnectivity-ca\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-modprobe-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-var-lib-kubelet\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cnibin\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.897782 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897781 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcp62\" (UniqueName: \"kubernetes.io/projected/9bc0c153-0125-47d4-bf2d-27668920eb0b-kube-api-access-tcp62\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897873 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-var-lib-kubelet\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897951 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-modprobe-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897960 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cnibin\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.897976 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.897982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-kubernetes\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-run\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.898042 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.398023452 +0000 UTC m=+2.236905160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898063 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-run\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-etc-tuned\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcgb\" (UniqueName: \"kubernetes.io/projected/d86f6ae9-bad4-41c8-be3c-61e4225a0501-kube-api-access-qvcgb\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898153 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-kubernetes\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-sys\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898167 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898159 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-env-overrides\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-sys-fs\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898240 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898261 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-sys-fs\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-config\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898326 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898333 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-sys\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898339 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft54x\" (UniqueName: \"kubernetes.io/projected/eb2cef57-0b45-422b-9390-ee103989900e-kube-api-access-ft54x\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7xz\" (UniqueName: \"kubernetes.io/projected/41738995-836d-4314-baa7-d2e7f5149a56-kube-api-access-rp7xz\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898371 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-konnectivity-ca\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvv6\" (UniqueName: \"kubernetes.io/projected/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-kube-api-access-pwvv6\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898431 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-script-lib\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898495 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-etc-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898526 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d86f6ae9-bad4-41c8-be3c-61e4225a0501-host\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-multus\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898557 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-etc-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898574 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-daemon-config\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.898959 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-netd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898605 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-multus\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898588 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d86f6ae9-bad4-41c8-be3c-61e4225a0501-host\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb80ad79-d9ca-41d8-a676-8844585591d4-iptables-alerter-script\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2cef57-0b45-422b-9390-ee103989900e-tmp-dir\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898662 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898680 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-netd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-systemd\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898769 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-config\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898782 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898789 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-systemd\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-device-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-node-log\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-agent-certs\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-hostroot\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898928 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2cef57-0b45-422b-9390-ee103989900e-tmp-dir\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.899726 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-k8s-cni-cncf-io\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898968 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-k8s-cni-cncf-io\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.898986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-multus-certs\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899030 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-node-log\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899032 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899034 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-multus-certs\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899072 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-hostroot\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899080 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovn-node-metrics-cert\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-conf-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-etc-kubernetes\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899151 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb80ad79-d9ca-41d8-a676-8844585591d4-iptables-alerter-script\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-dbus\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899164 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-conf-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899129 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-daemon-config\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-kubelet\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899236 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-dbus\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-bin\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.900611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899234 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-etc-kubernetes\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-system-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-socket-dir-parent\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899292 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899287 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-kubelet\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899341 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-bin\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899364 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899396 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-kubelet-config\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899402 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-system-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899366 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-socket-dir-parent\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-tmp\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.899433 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899455 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-kubelet-config\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899460 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-device-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-socket-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.901515 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.899482 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.399466991 +0000 UTC m=+2.238348683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899563 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-socket-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-slash\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-systemd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2cef57-0b45-422b-9390-ee103989900e-hosts-file\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-kubelet\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899696 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899748 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-slash\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899761 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2cef57-0b45-422b-9390-ee103989900e-hosts-file\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899781 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-systemd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-cnibin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899812 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-kubelet\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysconfig\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899839 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb80ad79-d9ca-41d8-a676-8844585591d4-host-slash\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899847 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-cnibin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-cni-binary-copy\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899887 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysconfig\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-systemd-units\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.902341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899924 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb80ad79-d9ca-41d8-a676-8844585591d4-host-slash\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zqr\" (UniqueName: \"kubernetes.io/projected/eb80ad79-d9ca-41d8-a676-8844585591d4-kube-api-access-s4zqr\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899947 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-conf\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.899985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-bin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-netns\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8vd\" (UniqueName: \"kubernetes.io/projected/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kube-api-access-fm8vd\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-var-lib-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900138 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-system-cni-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-registration-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-ovn\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnhk\" (UniqueName: \"kubernetes.io/projected/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-kube-api-access-vsnhk\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-host\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-netns\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d86f6ae9-bad4-41c8-be3c-61e4225a0501-serviceca\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-lib-modules\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f749v\" (UniqueName: \"kubernetes.io/projected/6b6edcdf-aa04-4afd-870c-4670f562dc06-kube-api-access-f749v\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-log-socket\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900671 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900763 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-conf\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900789 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-cni-binary-copy\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900803 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-bin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900839 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-netns\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900851 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-systemd-units\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.900987 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901067 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-lib-modules\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901127 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-system-cni-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901133 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-netns\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901177 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901199 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-var-lib-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.903722 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901202 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-registration-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901226 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-host\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901226 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901244 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-ovn\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901249 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-log-socket\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.901615 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d86f6ae9-bad4-41c8-be3c-61e4225a0501-serviceca\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.902730 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-etc-tuned\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.902889 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-tmp\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.902945 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovn-node-metrics-cert\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:57.904154 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.903406 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-agent-certs\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:57.954505 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.954486 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:57.954505 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.954503 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:57.954615 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.954512 2570 projected.go:196] Error preparing data for projected volume kube-api-access-w6z2d for pod openshift-network-diagnostics/network-check-target-ng9zm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:57.954615 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:57.954559 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d podName:f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.454544513 +0000 UTC m=+2.293426219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w6z2d" (UniqueName: "kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d") pod "network-check-target-ng9zm" (UID: "f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:57.956673 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.956655 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcgb\" (UniqueName: \"kubernetes.io/projected/d86f6ae9-bad4-41c8-be3c-61e4225a0501-kube-api-access-qvcgb\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:57.958260 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.958235 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcp62\" (UniqueName: \"kubernetes.io/projected/9bc0c153-0125-47d4-bf2d-27668920eb0b-kube-api-access-tcp62\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:41:57.959549 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.959517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft54x\" (UniqueName: \"kubernetes.io/projected/eb2cef57-0b45-422b-9390-ee103989900e-kube-api-access-ft54x\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:57.960553 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.960533 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvv6\" (UniqueName: \"kubernetes.io/projected/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-kube-api-access-pwvv6\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:57.960624 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.960560 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7xz\" (UniqueName: \"kubernetes.io/projected/41738995-836d-4314-baa7-d2e7f5149a56-kube-api-access-rp7xz\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:57.962404 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.962380 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8vd\" (UniqueName: \"kubernetes.io/projected/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kube-api-access-fm8vd\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:57.963089 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.963066 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zqr\" (UniqueName: \"kubernetes.io/projected/eb80ad79-d9ca-41d8-a676-8844585591d4-kube-api-access-s4zqr\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:57.963436 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.963418 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f749v\" (UniqueName: \"kubernetes.io/projected/6b6edcdf-aa04-4afd-870c-4670f562dc06-kube-api-access-f749v\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:57.963816 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:57.963798 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnhk\" (UniqueName: \"kubernetes.io/projected/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-kube-api-access-vsnhk\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:58.079918 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.079723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qg64f" Oct 30 21:41:58.112140 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.112119 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:41:58.116823 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.116801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:41:58.132646 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.132627 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:41:58.144736 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.144719 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:41:58.162540 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.162512 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0994e35dd707326725d576ce662475d4.slice/crio-84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1 WatchSource:0}: Error finding container 84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1: Status 404 returned error can't find the container with id 84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1 Oct 30 21:41:58.163344 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.163320 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb80ad79_d9ca_41d8_a676_8844585591d4.slice/crio-7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241 WatchSource:0}: Error finding container 7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241: Status 404 returned error can't find the container with id 7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241 Oct 30 21:41:58.163818 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.163790 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a99366fa1c27f76a4200901a9260b1.slice/crio-ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0 WatchSource:0}: Error finding container ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0: Status 404 returned error can't find the container with id ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0 Oct 30 21:41:58.164377 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.164349 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41738995_836d_4314_baa7_d2e7f5149a56.slice/crio-837d9efd319b03d70f7a396ec35fb19731bdb725e3e341792fb178650e1b6de5 WatchSource:0}: Error finding container 837d9efd319b03d70f7a396ec35fb19731bdb725e3e341792fb178650e1b6de5: Status 404 returned error can't find the container with id 837d9efd319b03d70f7a396ec35fb19731bdb725e3e341792fb178650e1b6de5 Oct 30 21:41:58.168730 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.167508 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:41:58.169702 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.169679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:41:58.178296 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.178275 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e31250d_f97f_4b8b_9836_2f77a5deb9ce.slice/crio-3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3 WatchSource:0}: Error finding container 3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3: Status 404 returned error can't find the container with id 3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3 Oct 30 21:41:58.178789 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.178773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:41:58.186731 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.186713 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3573ec_d74e_44d9_aec5_ca58f51e04fd.slice/crio-f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116 WatchSource:0}: Error finding container f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116: Status 404 returned error can't find the container with id f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116 Oct 30 21:41:58.190638 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.190623 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:41:58.195140 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.195121 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:41:58.198858 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.198836 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2cef57_0b45_422b_9390_ee103989900e.slice/crio-92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908 WatchSource:0}: Error finding container 92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908: Status 404 returned error can't find the container with id 92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908 Oct 30 21:41:58.203250 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.203232 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86f6ae9_bad4_41c8_be3c_61e4225a0501.slice/crio-980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555 WatchSource:0}: Error finding container 980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555: Status 404 returned error can't find the container with id 980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555 Oct 30 21:41:58.300643 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.300608 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:58.403849 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.403774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:58.403849 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.403812 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:58.404006 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.403903 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:58.404006 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.403905 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:58.404006 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.403957 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:41:59.403941606 +0000 UTC m=+3.242823311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:58.404006 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.403971 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:59.403964633 +0000 UTC m=+3.242846321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:58.504378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.504342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:41:58.504746 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.504472 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:58.504746 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.504486 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:58.504746 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.504495 2570 projected.go:196] Error preparing data for projected volume kube-api-access-w6z2d for pod openshift-network-diagnostics/network-check-target-ng9zm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:58.504746 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:58.504547 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d podName:f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:59.504529169 +0000 UTC m=+3.343410858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6z2d" (UniqueName: "kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d") pod "network-check-target-ng9zm" (UID: "f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:58.625471 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.625416 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tsn5z" event={"ID":"eb2cef57-0b45-422b-9390-ee103989900e","Type":"ContainerStarted","Data":"92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908"} Oct 30 21:41:58.626676 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.626651 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" event={"ID":"fe3573ec-d74e-44d9-aec5-ca58f51e04fd","Type":"ContainerStarted","Data":"f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116"} Oct 30 21:41:58.627790 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.627767 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerStarted","Data":"3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3"} Oct 30 21:41:58.628939 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.628910 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5kndd" event={"ID":"eb80ad79-d9ca-41d8-a676-8844585591d4","Type":"ContainerStarted","Data":"7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241"} Oct 30 21:41:58.629998 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.629975 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"99a99366fa1c27f76a4200901a9260b1","Type":"ContainerStarted","Data":"ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0"} Oct 30 21:41:58.630992 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.630972 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"0994e35dd707326725d576ce662475d4","Type":"ContainerStarted","Data":"84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1"} Oct 30 21:41:58.631994 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.631974 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" event={"ID":"41738995-836d-4314-baa7-d2e7f5149a56","Type":"ContainerStarted","Data":"837d9efd319b03d70f7a396ec35fb19731bdb725e3e341792fb178650e1b6de5"} Oct 30 21:41:58.633067 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.633047 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2r78l" event={"ID":"d86f6ae9-bad4-41c8-be3c-61e4225a0501","Type":"ContainerStarted","Data":"980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555"} Oct 30 21:41:58.725426 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.725360 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:58.853683 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.853630 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:58 +0000 UTC" deadline="2028-03-30 17:01:28.236954912 +0000 UTC" Oct 30 21:41:58.853683 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.853673 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12403h19m29.383284742s" Oct 30 21:41:58.948159 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.948138 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc0c153_0125_47d4_bf2d_27668920eb0b.slice/crio-d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362 WatchSource:0}: Error finding container d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362: Status 404 returned error can't find the container with id d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362 Oct 30 21:41:58.965092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:58.965065 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:58.973560 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.973535 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec68d72_5b2e_4f80_8ae2_4fddcdd3d55e.slice/crio-42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67 WatchSource:0}: Error finding container 42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67: Status 404 returned error can't find the container with id 42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67 Oct 30 21:41:58.974270 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:41:58.974249 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a71cf03_8c4c_421f_ad1a_dd8735c1b2fc.slice/crio-7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4 WatchSource:0}: Error finding container 7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4: Status 404 returned error can't find the container with id 7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4 Oct 30 21:41:59.413513 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.413429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:59.413900 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.413880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:59.414100 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.413804 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:59.414243 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.414232 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:01.414212335 +0000 UTC m=+5.253094040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:59.414394 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.414047 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:59.414534 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.414522 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:42:01.414504484 +0000 UTC m=+5.253386192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:59.515204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.515162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:41:59.515865 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.515411 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:59.515865 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.515435 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:59.515865 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.515448 2570 projected.go:196] Error preparing data for projected volume kube-api-access-w6z2d for pod openshift-network-diagnostics/network-check-target-ng9zm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:59.515865 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.515509 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d podName:f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:01.515491327 +0000 UTC m=+5.354373022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6z2d" (UniqueName: "kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d") pod "network-check-target-ng9zm" (UID: "f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:59.621104 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.620454 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:41:59.621104 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.620667 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:41:59.621104 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.620820 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:41:59.621104 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.620939 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:41:59.621104 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.621003 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:41:59.621104 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:41:59.621073 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:41:59.649341 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.649299 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cggst" event={"ID":"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc","Type":"ContainerStarted","Data":"7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4"} Oct 30 21:41:59.656785 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.656704 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67"} Oct 30 21:41:59.666706 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.666636 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qg64f" event={"ID":"9bc0c153-0125-47d4-bf2d-27668920eb0b","Type":"ContainerStarted","Data":"d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362"} Oct 30 21:41:59.854422 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.854353 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:58 +0000 UTC" deadline="2028-05-13 14:29:56.891127414 +0000 UTC" Oct 30 21:41:59.854422 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:41:59.854420 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13456h47m57.036711887s" Oct 30 21:42:01.433325 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:01.433267 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:01.434029 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:01.433370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:01.434029 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.433552 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:01.434029 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.433623 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:05.433603264 +0000 UTC m=+9.272484970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:01.434029 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.433695 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:01.434029 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.433731 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:42:05.433719315 +0000 UTC m=+9.272601008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:01.534858 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:01.534818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:01.535059 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.535022 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:42:01.535059 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.535043 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:42:01.535059 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.535057 2570 projected.go:196] Error preparing data for projected volume kube-api-access-w6z2d for pod openshift-network-diagnostics/network-check-target-ng9zm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:01.535260 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.535118 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d podName:f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:05.535100455 +0000 UTC m=+9.373982160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6z2d" (UniqueName: "kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d") pod "network-check-target-ng9zm" (UID: "f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:01.620473 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:01.620399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:01.620712 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.620551 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:01.620712 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:01.620593 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:01.620712 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.620666 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:01.620712 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:01.620701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:01.620928 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:01.620790 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:02.698770 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:02.698735 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerStarted","Data":"78d5e275c4b1c88461abba80394b2908591ff1f0790e436a42aa354e95017ffc"} Oct 30 21:42:03.620670 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.620396 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:03.620861 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:03.620786 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:03.620947 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.620869 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:03.621005 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:03.620951 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:03.621066 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.621015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:03.621121 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:03.621091 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:03.704857 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.704820 2570 generic.go:356] "Generic (PLEG): container finished" podID="99a99366fa1c27f76a4200901a9260b1" containerID="a794db469e508564fd6f94755b7e5c726affd9d03e242fb2fe8929aecfdd78e2" exitCode=0 Oct 30 21:42:03.705349 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.704981 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"99a99366fa1c27f76a4200901a9260b1","Type":"ContainerDied","Data":"a794db469e508564fd6f94755b7e5c726affd9d03e242fb2fe8929aecfdd78e2"} Oct 30 21:42:03.714961 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.714925 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="78d5e275c4b1c88461abba80394b2908591ff1f0790e436a42aa354e95017ffc" exitCode=0 Oct 30 21:42:03.715078 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:03.714980 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerDied","Data":"78d5e275c4b1c88461abba80394b2908591ff1f0790e436a42aa354e95017ffc"} Oct 30 21:42:04.722034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:04.722002 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"99a99366fa1c27f76a4200901a9260b1","Type":"ContainerStarted","Data":"874bdd3c96c25bb9bfca0c2410fa3eeae3dedb3f46adb5e760f3a45d1acf046a"} Oct 30 21:42:04.728233 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:04.728202 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cggst" event={"ID":"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc","Type":"ContainerStarted","Data":"9a98f3ce5b8a9d18a30948a6d87348edfdb91be180b448f4d583dd8d5cac0211"} Oct 30 21:42:04.737285 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:04.737202 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2r78l" event={"ID":"d86f6ae9-bad4-41c8-be3c-61e4225a0501","Type":"ContainerStarted","Data":"8231763efa87b1a2f4b94a0cb686fc8acefa6a47c9051d15052804e9f6cae54a"} Oct 30 21:42:04.741559 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:04.741531 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"0994e35dd707326725d576ce662475d4","Type":"ContainerStarted","Data":"4e59b3b4c41c22003c4febff248e308ca9626961ce2a80a8d4081868b078f88e"} Oct 30 21:42:04.845670 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:04.845603 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" podStartSLOduration=7.845585685 podStartE2EDuration="7.845585685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:04.844836573 +0000 UTC m=+8.683718287" watchObservedRunningTime="2026-10-30 21:42:04.845585685 +0000 UTC m=+8.684467400" Oct 30 21:42:04.980846 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:04.980776 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cggst" podStartSLOduration=3.192513615 podStartE2EDuration="7.980755995s" totalImagesPullingTime="4.78824238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:04.980586314 +0000 UTC m=+8.819468030" watchObservedRunningTime="2026-10-30 21:42:04.980755995 +0000 UTC m=+8.819637701" Oct 30 21:42:05.103486 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.103363 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" podStartSLOduration=8.103342732 podStartE2EDuration="8.103342732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:05.102434146 +0000 UTC m=+8.941315860" watchObservedRunningTime="2026-10-30 21:42:05.103342732 +0000 UTC m=+8.942224448" Oct 30 21:42:05.475509 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.475472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:05.475646 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.475557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:05.475730 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.475696 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:05.475794 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.475760 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.475739797 +0000 UTC m=+17.314621506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:05.475863 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.475833 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:05.475918 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.475873 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.475860272 +0000 UTC m=+17.314741981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:05.576482 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.576440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:05.576654 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.576616 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:42:05.576654 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.576644 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:42:05.576758 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.576657 2570 projected.go:196] Error preparing data for projected volume kube-api-access-w6z2d for pod openshift-network-diagnostics/network-check-target-ng9zm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:05.576758 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.576713 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d podName:f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:13.576695563 +0000 UTC m=+17.415577266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6z2d" (UniqueName: "kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d") pod "network-check-target-ng9zm" (UID: "f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:05.621486 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.621419 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:05.621641 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.621542 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:05.621641 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.621615 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:05.621758 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.621686 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:05.621758 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.621741 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:05.621856 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:05.621809 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:05.747228 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:05.747187 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" event={"ID":"fe3573ec-d74e-44d9-aec5-ca58f51e04fd","Type":"ContainerStarted","Data":"7eadc086660b641f56a3f86689de3e2bbff7c4110c795e266f6882bde5d73140"} Oct 30 21:42:06.633918 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:06.633481 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:42:06.634118 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:06.634000 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:42:06.753717 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:06.753659 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2r78l" podStartSLOduration=3.46722557 podStartE2EDuration="9.753639638s" totalImagesPullingTime="6.286414068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:05.225484201 +0000 UTC m=+9.064365917" watchObservedRunningTime="2026-10-30 21:42:06.753639638 +0000 UTC m=+10.592521349" Oct 30 21:42:07.504935 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.504872 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:42:07.548204 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.547991 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:07.504894014Z","UUID":"1d94b66f-6289-45b4-9db4-92ed73972de3","Handler":null,"Name":"","Endpoint":""} Oct 30 21:42:07.549812 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.549793 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:42:07.549938 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.549822 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:42:07.621202 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.621172 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:07.621401 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:07.621319 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:07.621461 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.621402 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:07.621507 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:07.621476 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:07.621560 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.621527 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:07.621606 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:07.621588 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:07.755994 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.755907 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" event={"ID":"fe3573ec-d74e-44d9-aec5-ca58f51e04fd","Type":"ContainerStarted","Data":"607e03c8bde963f61ddac4dc4f5fede2f896ea1d0b74911b706e85b17540ccae"} Oct 30 21:42:07.757820 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.757798 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:07.757913 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:07.757829 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tsn5z" event={"ID":"eb2cef57-0b45-422b-9390-ee103989900e","Type":"ContainerStarted","Data":"70625cb1d7015518786409ce29ff0ed48092a4102ae73df3672882e0a65130cd"} Oct 30 21:42:08.762754 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:08.762717 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qg64f" event={"ID":"9bc0c153-0125-47d4-bf2d-27668920eb0b","Type":"ContainerStarted","Data":"4357b583ca1c2925c9d7d791e68b72dab5339561bfb9de87c4d48dd475f413cd"} Oct 30 21:42:08.765454 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:08.765423 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="c030c8fb59fe1748fbd637c1750eacfbcf7316e0c524f53546b02140430f1811" exitCode=0 Oct 30 21:42:08.765587 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:08.765511 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerDied","Data":"c030c8fb59fe1748fbd637c1750eacfbcf7316e0c524f53546b02140430f1811"} Oct 30 21:42:08.769589 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:08.769562 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5kndd" event={"ID":"eb80ad79-d9ca-41d8-a676-8844585591d4","Type":"ContainerStarted","Data":"e0ea9081307468900a847aa759befa06d1eb86b56c49b2cd8287b5f5191ee72d"} Oct 30 21:42:08.884750 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:08.883758 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tsn5z" podStartSLOduration=2.565248079 podStartE2EDuration="11.883735543s" totalImagesPullingTime="9.318487464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:07.875748188 +0000 UTC m=+11.714629899" watchObservedRunningTime="2026-10-30 21:42:08.883735543 +0000 UTC m=+12.722617235" Oct 30 21:42:08.884750 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:08.883933 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qg64f" podStartSLOduration=3.19239089 podStartE2EDuration="11.883926956s" totalImagesPullingTime="8.691536066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:08.883820183 +0000 UTC m=+12.722701893" watchObservedRunningTime="2026-10-30 21:42:08.883926956 +0000 UTC m=+12.722808666" Oct 30 21:42:09.003071 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.003026 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5kndd" podStartSLOduration=3.016696292 podStartE2EDuration="12.003011583s" totalImagesPullingTime="8.986315291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:09.001626279 +0000 UTC m=+12.840507990" watchObservedRunningTime="2026-10-30 21:42:09.003011583 +0000 UTC m=+12.841893292" Oct 30 21:42:09.620810 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.620776 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:09.620994 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.620854 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:09.621051 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.621000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:09.621051 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:09.621005 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:09.621146 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:09.621096 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:09.621240 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:09.621206 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:09.776716 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.776474 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" event={"ID":"fe3573ec-d74e-44d9-aec5-ca58f51e04fd","Type":"ContainerStarted","Data":"43cd36783f1677d887d0c54ab64c73f2fec242e08d2d0e15f2ab5f9f0598ae60"} Oct 30 21:42:09.780350 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.780319 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" event={"ID":"41738995-836d-4314-baa7-d2e7f5149a56","Type":"ContainerStarted","Data":"b0f6be98cd1332f82222f753349d1ec7363989b61be1fed912f4f17cd67d26d8"} Oct 30 21:42:09.896957 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:09.896862 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" podStartSLOduration=2.656433831 podStartE2EDuration="12.896848535s" totalImagesPullingTime="10.240414704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:09.896696839 +0000 UTC m=+13.735578549" watchObservedRunningTime="2026-10-30 21:42:09.896848535 +0000 UTC m=+13.735730244" Oct 30 21:42:10.013700 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:10.013638 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" podStartSLOduration=1.535243973 podStartE2EDuration="13.013617533s" totalImagesPullingTime="11.47837356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:10.013406024 +0000 UTC m=+13.852287734" watchObservedRunningTime="2026-10-30 21:42:10.013617533 +0000 UTC m=+13.852499245" Oct 30 21:42:10.784522 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:10.784486 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="913c6b324709b148f01a2f910cb107059a9b04c0959ade9e35639a35774a0bc6" exitCode=0 Oct 30 21:42:10.785071 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:10.784578 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerDied","Data":"913c6b324709b148f01a2f910cb107059a9b04c0959ade9e35639a35774a0bc6"} Oct 30 21:42:11.620929 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:11.620897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:11.620929 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:11.620917 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:11.620929 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:11.620903 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:11.621177 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:11.621032 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:11.621177 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:11.621061 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:11.621177 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:11.621122 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:12.790943 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:12.790890 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="986042df4725cbc0c8608978d34b3b959d1db52e46cc57ef964515035337e34e" exitCode=0 Oct 30 21:42:12.790943 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:12.790945 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerDied","Data":"986042df4725cbc0c8608978d34b3b959d1db52e46cc57ef964515035337e34e"} Oct 30 21:42:13.536045 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:13.536003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:13.536234 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:13.536076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:13.536234 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.536177 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:13.536234 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.536182 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:13.536378 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.536246 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.536226695 +0000 UTC m=+33.375108395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:13.536378 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.536264 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.536254904 +0000 UTC m=+33.375136595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:13.620742 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:13.620715 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:13.620742 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:13.620732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:13.620970 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:13.620739 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:13.620970 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.620845 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:13.620970 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.620923 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:13.621086 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.620994 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:13.636892 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:13.636867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:13.637063 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.637047 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:42:13.637146 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.637070 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:42:13.637146 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.637083 2570 projected.go:196] Error preparing data for projected volume kube-api-access-w6z2d for pod openshift-network-diagnostics/network-check-target-ng9zm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:13.637146 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:13.637137 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d podName:f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.637119665 +0000 UTC m=+33.476001355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w6z2d" (UniqueName: "kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d") pod "network-check-target-ng9zm" (UID: "f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:14.798267 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:14.798095 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"85a2d7def8004497e997263733f946b21fa703888bcd05d80858380492a4a2db"} Oct 30 21:42:14.798717 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:14.798358 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"d0a5ba44488d3373bd4b97e955f67176aa8c53565a184fb9f058a30fd1a45a86"} Oct 30 21:42:15.620848 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.620678 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:15.621003 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.620747 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:15.621003 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:15.620929 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:15.621003 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.620755 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:15.621003 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:15.620990 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:15.621193 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:15.621079 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:15.803324 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.803278 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"0155e6723644342761a44ec14a2e56ee82711da15cc25b5ffe4cc08000e42f28"} Oct 30 21:42:15.803705 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.803361 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"ea454675c4730cff0fd771e4d53423313cbe465f7ae15428b6ae10a0541aad0b"} Oct 30 21:42:15.803705 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.803407 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"26975a607273649bb5ff9f138d90d47c9399c5672c94f050619122b573c31808"} Oct 30 21:42:15.803705 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.803440 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"e152874571a419635598948821ecbdf4960d661d781c2374be37a74bac83f94a"} Oct 30 21:42:15.805979 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.805954 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="c6c2ff618e38dd3a08c917a9d1c00612cd3d62548a0fe7d026021c90267f90ff" exitCode=0 Oct 30 21:42:15.806096 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:15.805990 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerDied","Data":"c6c2ff618e38dd3a08c917a9d1c00612cd3d62548a0fe7d026021c90267f90ff"} Oct 30 21:42:16.810242 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:16.810209 2570 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="531015aa0c422875110bad7b288888d082b648ca6510b1c8df537c017944c4bc" exitCode=0 Oct 30 21:42:16.810648 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:16.810263 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerDied","Data":"531015aa0c422875110bad7b288888d082b648ca6510b1c8df537c017944c4bc"} Oct 30 21:42:17.621069 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:17.621036 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:17.621256 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:17.621093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:17.621256 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:17.621098 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:17.621256 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:17.621175 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:17.621256 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:17.621252 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:17.621485 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:17.621342 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:17.816052 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:17.816023 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" event={"ID":"5e31250d-f97f-4b8b-9836-2f77a5deb9ce","Type":"ContainerStarted","Data":"cd3c0fac02a68e9fa10604c853814361ce979f106770464fd98b822f344f6ef5"} Oct 30 21:42:17.818995 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:17.818963 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"9c5561375856675a6ca6940cc7131e21c60268809b107cc922e88797ad577e97"} Oct 30 21:42:17.935874 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:17.935791 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8hwrm" podStartSLOduration=8.279305503 podStartE2EDuration="20.935778776s" totalImagesPullingTime="12.656473273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:17.935480224 +0000 UTC m=+21.774361931" watchObservedRunningTime="2026-10-30 21:42:17.935778776 +0000 UTC m=+21.774660479" Oct 30 21:42:18.379041 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:18.378961 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:42:18.379186 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:18.379061 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:18.379535 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:18.379515 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:42:19.620899 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.620874 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:19.621697 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.620966 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:19.621697 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:19.620995 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:19.621697 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.621057 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:19.621697 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:19.621056 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:19.621697 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:19.621130 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:19.827733 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.827703 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" event={"ID":"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e","Type":"ContainerStarted","Data":"a08fc1442b220c1e2c62283046ee599554cf495b687f1ea0276813ad8e5375d4"} Oct 30 21:42:19.828097 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.828077 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:42:19.828228 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.828145 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:42:19.828228 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.828191 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:42:19.846817 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.846794 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:42:19.849843 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:19.849824 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:42:20.200396 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:20.200350 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:20.223479 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:20.223434 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" podStartSLOduration=7.548542236 podStartE2EDuration="23.223421146s" totalImagesPullingTime="15.67487891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:20.098910006 +0000 UTC m=+23.937791715" watchObservedRunningTime="2026-10-30 21:42:20.223421146 +0000 UTC m=+24.062302856" Oct 30 21:42:20.444721 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:20.444687 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:21.621121 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.620941 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:21.621584 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.620949 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:21.621584 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:21.621194 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:21.621584 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.620991 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:21.621584 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:21.621280 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:21.621584 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:21.621405 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:21.932145 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.932096 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpfn7"] Oct 30 21:42:21.932345 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.932224 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:21.932423 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:21.932361 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:21.932742 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.932721 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ng9zm"] Oct 30 21:42:21.932830 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.932796 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:21.932909 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:21.932889 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:21.980821 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.980797 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-56vhn"] Oct 30 21:42:21.980903 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:21.980862 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:21.980942 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:21.980924 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:23.171651 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.171614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:23.342904 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.342879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:23.620926 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.620894 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:23.621088 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.620921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:23.621088 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.621002 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:23.621167 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:23.621144 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:23.621245 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:23.621229 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:23.621335 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:23.621300 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:23.660984 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.660954 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:23.831745 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:23.831714 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:25.620325 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:25.620273 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:25.620325 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:25.620298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:25.620911 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:25.620285 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:25.620911 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:25.620390 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:25.620911 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:25.620473 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:25.620911 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:25.620535 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:27.038329 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.038264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.266374 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.266346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.445477 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.445437 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.618774 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.618739 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.620715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.620691 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:27.620715 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.620712 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:27.620865 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.620794 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:27.620865 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:27.620811 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpfn7" podUID="6b6edcdf-aa04-4afd-870c-4670f562dc06" Oct 30 21:42:27.620865 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:27.620858 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ng9zm" podUID="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" Oct 30 21:42:27.620979 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:27.620929 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56vhn" podUID="04ecd634-c544-41e3-82bd-b5e36a5d57cd" Oct 30 21:42:27.747648 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.747568 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeReady" Oct 30 21:42:27.747807 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.747719 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:42:27.804532 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.804499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:27.844098 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.844071 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nwqgt"] Oct 30 21:42:27.850697 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.850675 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:27.861336 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.861317 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jfjmh"] Oct 30 21:42:27.864235 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.864215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:27.867645 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.866930 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lbksg"] Oct 30 21:42:27.871630 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.871613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:27.906690 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.906665 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nwqgt"] Oct 30 21:42:27.907151 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.907117 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:42:27.907405 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.907378 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6j5v7\"" Oct 30 21:42:27.908061 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.908045 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:42:27.926410 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.926387 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jfjmh"] Oct 30 21:42:27.933730 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxq22\" (UniqueName: \"kubernetes.io/projected/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-api-access-dxq22\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:27.933839 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:27.933839 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:27.933839 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r959h\" (UniqueName: \"kubernetes.io/projected/8bacb62d-563f-428d-9bf1-25b14d6e59c0-kube-api-access-r959h\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:27.933984 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm2b\" (UniqueName: \"kubernetes.io/projected/b2d2e278-5632-4d36-87d8-69023ae58aa0-kube-api-access-rwm2b\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:27.933984 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d2e278-5632-4d36-87d8-69023ae58aa0-cert\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:27.933984 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ea24ed-9083-4214-ac43-06a49f4611dc-data-volume\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:27.933984 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933957 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:27.934098 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.933999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bacb62d-563f-428d-9bf1-25b14d6e59c0-config-volume\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:27.934098 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.934058 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48ea24ed-9083-4214-ac43-06a49f4611dc-crio-socket\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:27.934155 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.934099 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bacb62d-563f-428d-9bf1-25b14d6e59c0-tmp-dir\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:27.973298 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:27.973274 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lbksg"] Oct 30 21:42:28.006967 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.006887 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:42:28.007124 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.007108 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:42:28.007222 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.007209 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tpd8h\"" Oct 30 21:42:28.007254 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.007234 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbz5v\"" Oct 30 21:42:28.011713 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.011697 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:42:28.011804 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.011742 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:42:28.011804 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.011771 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:42:28.011804 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.011774 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:42:28.011804 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.011800 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:42:28.034767 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034744 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r959h\" (UniqueName: \"kubernetes.io/projected/8bacb62d-563f-428d-9bf1-25b14d6e59c0-kube-api-access-r959h\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.034829 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm2b\" (UniqueName: \"kubernetes.io/projected/b2d2e278-5632-4d36-87d8-69023ae58aa0-kube-api-access-rwm2b\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:28.034829 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d2e278-5632-4d36-87d8-69023ae58aa0-cert\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:28.034829 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034810 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ea24ed-9083-4214-ac43-06a49f4611dc-data-volume\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.034829 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.035005 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bacb62d-563f-428d-9bf1-25b14d6e59c0-config-volume\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.035005 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.034880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48ea24ed-9083-4214-ac43-06a49f4611dc-crio-socket\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.035005 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.034942 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:28.035005 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.035006 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.534988937 +0000 UTC m=+32.373870625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : configmap "kube-rbac-proxy" not found Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bacb62d-563f-428d-9bf1-25b14d6e59c0-tmp-dir\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxq22\" (UniqueName: \"kubernetes.io/projected/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-api-access-dxq22\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035137 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035150 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ea24ed-9083-4214-ac43-06a49f4611dc-data-volume\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035191 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48ea24ed-9083-4214-ac43-06a49f4611dc-crio-socket\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.035232 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.035268 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.035283 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls podName:8bacb62d-563f-428d-9bf1-25b14d6e59c0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.535269812 +0000 UTC m=+32.374151520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls") pod "dns-default-nwqgt" (UID: "8bacb62d-563f-428d-9bf1-25b14d6e59c0") : secret "dns-default-metrics-tls" not found Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.035338 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.535298859 +0000 UTC m=+32.374180547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:28.035394 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035344 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bacb62d-563f-428d-9bf1-25b14d6e59c0-tmp-dir\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.035705 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.035522 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bacb62d-563f-428d-9bf1-25b14d6e59c0-config-volume\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.039053 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.039037 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d2e278-5632-4d36-87d8-69023ae58aa0-cert\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:28.092148 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.092120 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r959h\" (UniqueName: \"kubernetes.io/projected/8bacb62d-563f-428d-9bf1-25b14d6e59c0-kube-api-access-r959h\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.092294 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.092255 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxq22\" (UniqueName: \"kubernetes.io/projected/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-api-access-dxq22\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.092407 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.092391 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm2b\" (UniqueName: \"kubernetes.io/projected/b2d2e278-5632-4d36-87d8-69023ae58aa0-kube-api-access-rwm2b\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:28.175626 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.175600 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:42:28.539378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.539329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.539378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.539381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.539435 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.539508 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.539521 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.539543 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.539578 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.53956183 +0000 UTC m=+33.378443519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : configmap "kube-rbac-proxy" not found Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.539597 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.539587935 +0000 UTC m=+33.378469626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:28.539612 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:28.539610 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls podName:8bacb62d-563f-428d-9bf1-25b14d6e59c0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:29.539603309 +0000 UTC m=+33.378485002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls") pod "dns-default-nwqgt" (UID: "8bacb62d-563f-428d-9bf1-25b14d6e59c0") : secret "dns-default-metrics-tls" not found Oct 30 21:42:28.629418 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.629390 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jfjmh"] Oct 30 21:42:28.632072 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.632049 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7qf6l"] Oct 30 21:42:28.633685 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:28.633660 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d2e278_5632_4d36_87d8_69023ae58aa0.slice/crio-4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db WatchSource:0}: Error finding container 4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db: Status 404 returned error can't find the container with id 4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db Oct 30 21:42:28.657988 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.657969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.693089 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.692979 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7qf6l"] Oct 30 21:42:28.710351 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.710327 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:42:28.710426 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.710352 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:42:28.710426 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.710363 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:42:28.710426 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.710370 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:42:28.710426 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.710406 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-77bv7\"" Oct 30 21:42:28.741531 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.741507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ht2x\" (UniqueName: \"kubernetes.io/projected/b59e3ce2-a114-414a-bfdd-23a58df02c91-kube-api-access-6ht2x\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.741615 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.741548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-key\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.741680 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.741665 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.842902 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.842870 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.843058 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.842934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ht2x\" (UniqueName: \"kubernetes.io/projected/b59e3ce2-a114-414a-bfdd-23a58df02c91-kube-api-access-6ht2x\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.843058 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.842958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-key\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.843611 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.843589 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.847180 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.845556 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-key\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.849515 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.849494 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jfjmh" event={"ID":"b2d2e278-5632-4d36-87d8-69023ae58aa0","Type":"ContainerStarted","Data":"4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db"} Oct 30 21:42:28.872140 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.872113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:28.899903 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.899860 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ht2x\" (UniqueName: \"kubernetes.io/projected/b59e3ce2-a114-414a-bfdd-23a58df02c91-kube-api-access-6ht2x\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:28.982977 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:28.982957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:42:29.050816 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.049096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:29.325820 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.325794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:29.368254 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.368205 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-7qf6l"] Oct 30 21:42:29.371800 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:29.371778 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59e3ce2_a114_414a_bfdd_23a58df02c91.slice/crio-f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10 WatchSource:0}: Error finding container f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10: Status 404 returned error can't find the container with id f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10 Oct 30 21:42:29.548050 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.548011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:29.548232 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.548056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:29.548232 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.548085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:29.548232 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.548129 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:29.548232 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.548167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:29.548232 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548176 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548241 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.548226801 +0000 UTC m=+35.387108490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548243 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548176 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548270 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548296 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls podName:8bacb62d-563f-428d-9bf1-25b14d6e59c0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.548285606 +0000 UTC m=+35.387167311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls") pod "dns-default-nwqgt" (UID: "8bacb62d-563f-428d-9bf1-25b14d6e59c0") : secret "dns-default-metrics-tls" not found Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548248 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548339 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.548320672 +0000 UTC m=+35.387202374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : configmap "kube-rbac-proxy" not found Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548356 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs podName:6b6edcdf-aa04-4afd-870c-4670f562dc06 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:01.548349367 +0000 UTC m=+65.387231055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs") pod "network-metrics-daemon-bpfn7" (UID: "6b6edcdf-aa04-4afd-870c-4670f562dc06") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:29.548453 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:29.548369 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret podName:04ecd634-c544-41e3-82bd-b5e36a5d57cd nodeName:}" failed. No retries permitted until 2026-10-30 21:43:01.548363494 +0000 UTC m=+65.387245181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret") pod "global-pull-secret-syncer-56vhn" (UID: "04ecd634-c544-41e3-82bd-b5e36a5d57cd") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:29.621194 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.621127 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:42:29.621378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.621204 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:42:29.621378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.621229 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:29.648918 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.648895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:29.674237 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.674220 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:42:29.674366 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.674219 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:42:29.674366 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.674278 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-msl6j\"" Oct 30 21:42:29.674366 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.674291 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcz6k\"" Oct 30 21:42:29.674366 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.674325 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:42:29.681927 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.681906 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:29.696686 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.696667 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:29.708429 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.708409 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-spllf\"" Oct 30 21:42:29.708429 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.708413 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:42:29.852954 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:29.852923 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" event={"ID":"b59e3ce2-a114-414a-bfdd-23a58df02c91","Type":"ContainerStarted","Data":"f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10"} Oct 30 21:42:30.091127 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:30.091051 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ng9zm"] Oct 30 21:42:30.098719 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:30.098689 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b71dff_9f5e_4722_8ee4_eb5dc2a2ca2b.slice/crio-81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133 WatchSource:0}: Error finding container 81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133: Status 404 returned error can't find the container with id 81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133 Oct 30 21:42:30.856897 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:30.856856 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ng9zm" event={"ID":"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b","Type":"ContainerStarted","Data":"81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133"} Oct 30 21:42:31.564613 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:31.564581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:31.564668 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:31.564691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:31.564835 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:31.564888 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls podName:8bacb62d-563f-428d-9bf1-25b14d6e59c0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.564873241 +0000 UTC m=+39.403754932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls") pod "dns-default-nwqgt" (UID: "8bacb62d-563f-428d-9bf1-25b14d6e59c0") : secret "dns-default-metrics-tls" not found Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:31.564928 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:31.564962 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.564952628 +0000 UTC m=+39.403834316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : configmap "kube-rbac-proxy" not found Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:31.565010 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:31.565102 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:31.565032 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.565024808 +0000 UTC m=+39.403906496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:31.964948 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:31.964893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.293765 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:32.293695 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:32.863897 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:32.863865 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jfjmh" event={"ID":"b2d2e278-5632-4d36-87d8-69023ae58aa0","Type":"ContainerStarted","Data":"8004a0fcac5e179ffa5239c02e8eebbad83a7c98e19ff7025ef72a0256577413"} Oct 30 21:42:32.866076 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:32.866049 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" event={"ID":"b59e3ce2-a114-414a-bfdd-23a58df02c91","Type":"ContainerStarted","Data":"23ae10c3d02ed3c06dd5c36d106e715904fcce85e571e377349aa9e2f258e3c3"} Oct 30 21:42:32.983839 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:32.981514 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jfjmh" podStartSLOduration=1.917983574 podStartE2EDuration="5.981496981s" totalImagesPullingTime="4.063513407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:32.980490554 +0000 UTC m=+36.819372263" watchObservedRunningTime="2026-10-30 21:42:32.981496981 +0000 UTC m=+36.820378691" Oct 30 21:42:33.094823 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:33.094482 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" podStartSLOduration=1.780699363 podStartE2EDuration="5.09446287s" totalImagesPullingTime="3.313763507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:33.094233536 +0000 UTC m=+36.933115265" watchObservedRunningTime="2026-10-30 21:42:33.09446287 +0000 UTC m=+36.933344578" Oct 30 21:42:34.061664 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.061620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.242986 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.242937 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.572892 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.572858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.748218 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.748191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.873487 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.873414 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ng9zm" event={"ID":"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b","Type":"ContainerStarted","Data":"e6fc0aadf92cee8c158c82d94b01b97376709985e31fa92a703464a32bd97639"} Oct 30 21:42:34.873622 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.873547 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:42:34.987166 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.987133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:34.989467 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:34.989430 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ng9zm" podStartSLOduration=33.465334064 podStartE2EDuration="37.989418415s" totalImagesPullingTime="4.524084351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:34.989085646 +0000 UTC m=+38.827967358" watchObservedRunningTime="2026-10-30 21:42:34.989418415 +0000 UTC m=+38.828300115" Oct 30 21:42:35.600548 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:35.600514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:35.600548 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:35.600552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:35.601079 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:35.600655 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:35.601079 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:35.600664 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:35.601079 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:35.600697 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls podName:8bacb62d-563f-428d-9bf1-25b14d6e59c0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:43.600685744 +0000 UTC m=+47.439567460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls") pod "dns-default-nwqgt" (UID: "8bacb62d-563f-428d-9bf1-25b14d6e59c0") : secret "dns-default-metrics-tls" not found Oct 30 21:42:35.601079 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:35.600710 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:43.600704422 +0000 UTC m=+47.439586109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:35.601079 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:35.600663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:35.601337 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:35.601211 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:36.054389 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.054317 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tsn5z_eb2cef57-0b45-422b-9390-ee103989900e/dns-node-resolver/0.log" Oct 30 21:42:36.112635 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.112599 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.310367 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.310269 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.504658 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.504626 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.651250 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.651222 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2r78l_d86f6ae9-bad4-41c8-be3c-61e4225a0501/node-ca/0.log" Oct 30 21:42:36.655634 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.655614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.814903 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.814869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:36.970265 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:36.970199 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:37.852772 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:37.852746 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jfjmh_b2d2e278-5632-4d36-87d8-69023ae58aa0/serve-healthcheck-canary/0.log" Oct 30 21:42:38.102224 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:38.102188 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:38.252005 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:38.251916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:38.405645 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:38.405611 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:38.564666 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:38.564591 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:38.720153 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:38.720120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:38.893636 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:38.893601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.108205 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:40.108175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.315320 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:40.315272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.506111 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:40.506033 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.707058 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:40.707026 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.900829 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:40.900796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:41.249013 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:41.248947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:42.136401 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:42.136369 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:42.338918 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:42.338887 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:42.517687 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:42.517613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:42.715535 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:42.715504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:42.911428 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:42.911396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:43.656361 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:43.656329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:43.656361 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:43.656361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:43.656889 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:43.656480 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:43.656889 ip-10-0-134-73 kubenswrapper[2570]: E1030 21:42:43.656579 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls podName:48ea24ed-9083-4214-ac43-06a49f4611dc nodeName:}" failed. No retries permitted until 2026-10-30 21:42:59.65655593 +0000 UTC m=+63.495437636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lbksg" (UID: "48ea24ed-9083-4214-ac43-06a49f4611dc") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:43.658744 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:43.658714 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:43.765465 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:43.765436 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:44.110572 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.110531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.134532 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.134505 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nwqgt"] Oct 30 21:42:44.138319 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:44.138267 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bacb62d_563f_428d_9bf1_25b14d6e59c0.slice/crio-24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c WatchSource:0}: Error finding container 24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c: Status 404 returned error can't find the container with id 24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c Oct 30 21:42:44.265619 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.265592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.417894 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.417786 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.583386 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.583296 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.754189 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.754103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:44.901137 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.901105 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwqgt" event={"ID":"8bacb62d-563f-428d-9bf1-25b14d6e59c0","Type":"ContainerStarted","Data":"24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c"} Oct 30 21:42:44.909003 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:44.908955 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:46.907971 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:46.907936 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwqgt" event={"ID":"8bacb62d-563f-428d-9bf1-25b14d6e59c0","Type":"ContainerStarted","Data":"589e818a4c08c1c948f77ad64e09e1df687ec3ab06bbba536d92f4721f357ded"} Oct 30 21:42:46.908406 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:46.907985 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nwqgt" event={"ID":"8bacb62d-563f-428d-9bf1-25b14d6e59c0","Type":"ContainerStarted","Data":"547aa1de3455b9c8436712619ed50900760181ee13d9fd70e20426be4780320f"} Oct 30 21:42:46.908406 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:46.908073 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:47.021870 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.021822 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-nwqgt" podStartSLOduration=17.87454201 podStartE2EDuration="20.02180915s" totalImagesPullingTime="2.14726714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:47.020997357 +0000 UTC m=+50.859879078" watchObservedRunningTime="2026-10-30 21:42:47.02180915 +0000 UTC m=+50.860690860" Oct 30 21:42:47.206543 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.206468 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.371293 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.371262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.532034 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.531959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.746529 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.746497 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:47.909092 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.909056 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f7d486fcd-vl4dg"] Oct 30 21:42:47.910940 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:47.910921 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.050465 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.050427 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.071405 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.071383 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:42:48.071405 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.071384 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:42:48.071603 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.071388 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:42:48.071603 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.071452 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:42:48.071603 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.071526 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hln5f\"" Oct 30 21:42:48.072664 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.072646 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f7d486fcd-vl4dg"] Oct 30 21:42:48.073340 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.073325 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-djc8v"] Oct 30 21:42:48.075138 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.075123 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.090038 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-image-registry-private-configuration\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090146 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-registry-certificates\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090146 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70d8105d-db42-4ae1-9265-028ab1488148-ca-trust-extracted\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090258 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-registry-tls\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090258 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-installation-pull-secrets\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090258 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-bound-sa-token\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090395 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-trusted-ca\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.090449 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.090389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2dz\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-kube-api-access-hq2dz\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.121569 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.121548 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-djc8v"] Oct 30 21:42:48.127652 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.127634 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:42:48.127652 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.127636 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:42:48.127939 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.127925 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-4t47c\"" Oct 30 21:42:48.190938 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.190883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2dz\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-kube-api-access-hq2dz\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.190938 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.190917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-tmp\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.191082 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.190970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-image-registry-private-configuration\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191082 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-registry-certificates\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191082 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191073 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70d8105d-db42-4ae1-9265-028ab1488148-ca-trust-extracted\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191251 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-registry-tls\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191251 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-installation-pull-secrets\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191251 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-bound-sa-token\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191444 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-trusted-ca\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.191444 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.191390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5f8\" (UniqueName: \"kubernetes.io/projected/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-kube-api-access-6m5f8\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.192069 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.192048 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70d8105d-db42-4ae1-9265-028ab1488148-ca-trust-extracted\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.192674 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.192633 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-registry-certificates\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.196044 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.192771 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-trusted-ca\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.196044 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.195025 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-installation-pull-secrets\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.196802 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.196779 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-image-registry-private-configuration\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.197772 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.197740 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-registry-tls\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.224519 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.224499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.247960 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.247941 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-bound-sa-token\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.248284 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.248264 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2dz\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-kube-api-access-hq2dz\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.292139 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.292120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5f8\" (UniqueName: \"kubernetes.io/projected/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-kube-api-access-6m5f8\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.292215 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.292148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-tmp\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.292454 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.292440 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-tmp\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.348542 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.348511 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5f8\" (UniqueName: \"kubernetes.io/projected/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-kube-api-access-6m5f8\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.384688 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.384663 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:42:48.469487 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.469427 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.520521 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.520479 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:48.650930 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.650902 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:48.801164 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.801108 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-djc8v"] Oct 30 21:42:48.803887 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:48.803864 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cbfbcf_0c69_4649_a966_c29bb2ef0d05.slice/crio-7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a WatchSource:0}: Error finding container 7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a: Status 404 returned error can't find the container with id 7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a Oct 30 21:42:48.903692 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.903667 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f7d486fcd-vl4dg"] Oct 30 21:42:48.907198 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:48.907176 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d8105d_db42_4ae1_9265_028ab1488148.slice/crio-13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2 WatchSource:0}: Error finding container 13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2: Status 404 returned error can't find the container with id 13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2 Oct 30 21:42:48.912911 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.912891 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" event={"ID":"70d8105d-db42-4ae1-9265-028ab1488148","Type":"ContainerStarted","Data":"13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2"} Oct 30 21:42:48.914077 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:48.914058 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-djc8v" event={"ID":"03cbfbcf-0c69-4649-a966-c29bb2ef0d05","Type":"ContainerStarted","Data":"7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a"} Oct 30 21:42:49.053520 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:49.053448 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:49.226704 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:49.226675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:49.656088 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:49.656054 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:49.906162 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:49.906035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:49.919643 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:49.919599 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" event={"ID":"70d8105d-db42-4ae1-9265-028ab1488148","Type":"ContainerStarted","Data":"7fb7ce87c462028006f3dde96a3125e94b945de0f81c0105702038baa81e880a"} Oct 30 21:42:49.920026 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:49.919679 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:42:50.038570 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:50.038506 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" podStartSLOduration=3.038484874 podStartE2EDuration="3.038484874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:50.037715132 +0000 UTC m=+53.876596871" watchObservedRunningTime="2026-10-30 21:42:50.038484874 +0000 UTC m=+53.877366586" Oct 30 21:42:50.067079 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:50.067056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:51.844774 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:51.844747 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:42:52.008018 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:52.007977 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.190058 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:52.189953 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.357695 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:52.357649 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.521405 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:52.521326 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.690235 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:52.690201 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:52.908535 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:52.908506 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.109010 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.108977 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.695059 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.695017 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.765742 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.765711 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5"] Oct 30 21:42:53.770044 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.770023 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:53.833923 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.833891 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cb39913-45bd-4e03-8a53-e5a269238d4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cvlq5\" (UID: \"6cb39913-45bd-4e03-8a53-e5a269238d4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:53.871020 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.870993 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:42:53.871259 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.871232 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s2czt\"" Oct 30 21:42:53.872620 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.872590 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5"] Oct 30 21:42:53.918924 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.918893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:53.934615 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.934589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cb39913-45bd-4e03-8a53-e5a269238d4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cvlq5\" (UID: \"6cb39913-45bd-4e03-8a53-e5a269238d4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:53.937375 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:53.937351 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cb39913-45bd-4e03-8a53-e5a269238d4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cvlq5\" (UID: \"6cb39913-45bd-4e03-8a53-e5a269238d4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:54.082444 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:54.082413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:54.457460 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:54.457337 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5"] Oct 30 21:42:54.460758 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:54.460733 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb39913_45bd_4e03_8a53_e5a269238d4e.slice/crio-bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2 WatchSource:0}: Error finding container bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2: Status 404 returned error can't find the container with id bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2 Oct 30 21:42:54.934648 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:54.934607 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" event={"ID":"6cb39913-45bd-4e03-8a53-e5a269238d4e","Type":"ContainerStarted","Data":"bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2"} Oct 30 21:42:56.912383 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:56.912346 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nwqgt" Oct 30 21:42:56.942628 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:56.942570 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" event={"ID":"6cb39913-45bd-4e03-8a53-e5a269238d4e","Type":"ContainerStarted","Data":"0b3e51d588d0fb0848760a772a962465e2671a6a2a75d3d56bc295c2855f1bf8"} Oct 30 21:42:56.942787 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:56.942743 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:56.948102 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:56.948079 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:42:57.154575 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:57.154512 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" podStartSLOduration=2.561138593 podStartE2EDuration="4.154493409s" totalImagesPullingTime="1.593354816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:57.153527774 +0000 UTC m=+60.992409510" watchObservedRunningTime="2026-10-30 21:42:57.154493409 +0000 UTC m=+60.993375121" Oct 30 21:42:58.020315 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.020258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.178548 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.178510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.332573 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.332546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.387806 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.387771 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm"] Oct 30 21:42:58.389878 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.389854 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.442078 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.442046 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:42:58.442239 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.442224 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:42:58.442290 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.442266 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:42:58.442370 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.442350 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:42:58.442450 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.442436 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pdrmb\"" Oct 30 21:42:58.474286 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.474257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.474450 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.474334 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9kb\" (UniqueName: \"kubernetes.io/projected/50ebddb7-2e87-4a26-844e-bcb12013c7c2-kube-api-access-8q9kb\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.474520 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.474451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ebddb7-2e87-4a26-844e-bcb12013c7c2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.474520 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.474500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.491208 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.491187 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:42:58.492392 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.492368 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm"] Oct 30 21:42:58.496258 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.496229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:58.575551 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.575516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9kb\" (UniqueName: \"kubernetes.io/projected/50ebddb7-2e87-4a26-844e-bcb12013c7c2-kube-api-access-8q9kb\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.575724 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.575584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ebddb7-2e87-4a26-844e-bcb12013c7c2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.575724 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.575624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.575724 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.575666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.576362 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.576337 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ebddb7-2e87-4a26-844e-bcb12013c7c2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.578476 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.578452 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.578596 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.578476 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.633682 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.633600 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9kb\" (UniqueName: \"kubernetes.io/projected/50ebddb7-2e87-4a26-844e-bcb12013c7c2-kube-api-access-8q9kb\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:58.702432 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:58.702398 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:42:59.075981 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:59.075950 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm"] Oct 30 21:42:59.110152 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:42:59.110119 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ebddb7_2e87_4a26_844e_bcb12013c7c2.slice/crio-4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02 WatchSource:0}: Error finding container 4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02: Status 404 returned error can't find the container with id 4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02 Oct 30 21:42:59.686547 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:59.686505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:59.689432 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:59.689393 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:42:59.953987 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:42:59.953905 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" event={"ID":"50ebddb7-2e87-4a26-844e-bcb12013c7c2","Type":"ContainerStarted","Data":"4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02"} Oct 30 21:43:00.038522 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:00.038488 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbz5v\"" Oct 30 21:43:00.048552 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:00.048518 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:00.430487 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:00.430462 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lbksg"] Oct 30 21:43:00.433537 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:43:00.433510 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ea24ed_9083_4214_ac43_06a49f4611dc.slice/crio-8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd WatchSource:0}: Error finding container 8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd: Status 404 returned error can't find the container with id 8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd Oct 30 21:43:00.961055 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:00.961027 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" event={"ID":"50ebddb7-2e87-4a26-844e-bcb12013c7c2","Type":"ContainerStarted","Data":"4bc848eaf3ba63cb160a991895222d9181d34a250ccbd318bfe3bcdcc94066c3"} Oct 30 21:43:00.963027 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:00.962999 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbksg" event={"ID":"48ea24ed-9083-4214-ac43-06a49f4611dc","Type":"ContainerStarted","Data":"aae7c8dee9b76325d80f9b85470ed79709c86389b972b818fa130cbabd7f9da6"} Oct 30 21:43:00.963126 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:00.963061 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbksg" event={"ID":"48ea24ed-9083-4214-ac43-06a49f4611dc","Type":"ContainerStarted","Data":"8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd"} Oct 30 21:43:01.602320 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.602274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:01.602761 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.602364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:01.702604 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.702567 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:43:01.702604 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.702567 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:43:01.715430 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.715399 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:01.715567 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.715400 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:01.821381 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.821345 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-spllf\"" Oct 30 21:43:01.831537 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.831509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:01.833582 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.833557 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcz6k\"" Oct 30 21:43:01.843694 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.843664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:01.968408 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.968369 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" event={"ID":"50ebddb7-2e87-4a26-844e-bcb12013c7c2","Type":"ContainerStarted","Data":"9deace123a6af46b7dd9315704c6682779758b44446c48b3c3fd71c13fdbc37c"} Oct 30 21:43:01.970467 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:01.970442 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbksg" event={"ID":"48ea24ed-9083-4214-ac43-06a49f4611dc","Type":"ContainerStarted","Data":"00f70c943265c9dc031d93dde8d29674a0e0f59bdec9f73be06f4d6f3601caf4"} Oct 30 21:43:02.085795 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:02.085683 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" podStartSLOduration=2.336744122 podStartE2EDuration="4.085664938s" totalImagesPullingTime="1.748920816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:02.084184169 +0000 UTC m=+65.923065901" watchObservedRunningTime="2026-10-30 21:43:02.085664938 +0000 UTC m=+65.924546650" Oct 30 21:43:02.206079 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:02.206032 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpfn7"] Oct 30 21:43:02.209373 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:43:02.209341 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6edcdf_aa04_4afd_870c_4670f562dc06.slice/crio-442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd WatchSource:0}: Error finding container 442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd: Status 404 returned error can't find the container with id 442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd Oct 30 21:43:02.255676 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:02.255644 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-56vhn"] Oct 30 21:43:02.259323 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:43:02.259271 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ecd634_c544_41e3_82bd_b5e36a5d57cd.slice/crio-9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf WatchSource:0}: Error finding container 9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf: Status 404 returned error can't find the container with id 9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf Oct 30 21:43:02.975643 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:02.975596 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpfn7" event={"ID":"6b6edcdf-aa04-4afd-870c-4670f562dc06","Type":"ContainerStarted","Data":"442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd"} Oct 30 21:43:02.977189 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:02.977143 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-56vhn" event={"ID":"04ecd634-c544-41e3-82bd-b5e36a5d57cd","Type":"ContainerStarted","Data":"9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf"} Oct 30 21:43:03.515326 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:03.515274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:03.678585 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:03.678551 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:03.835188 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:03.835153 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:04.253402 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.253367 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:04.600247 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.600185 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:04.801478 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.801416 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:04.827070 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.827042 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6"] Oct 30 21:43:04.851369 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.850618 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:04.889662 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.889630 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6"] Oct 30 21:43:04.928709 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.928448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:04.928709 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.928503 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:04.928709 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.928533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4nvs\" (UniqueName: \"kubernetes.io/projected/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-kube-api-access-p4nvs\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:04.928709 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.928567 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:04.937252 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.937227 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:43:04.937395 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.937343 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dzj62\"" Oct 30 21:43:04.951680 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.948884 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:43:04.951680 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.950594 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qmc6j"] Oct 30 21:43:04.971175 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.971151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:04.973430 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:04.973409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.010086 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.009874 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpfn7" event={"ID":"6b6edcdf-aa04-4afd-870c-4670f562dc06","Type":"ContainerStarted","Data":"cbef81c0cc530db432a6dad58858190679420949494cca4052bf3e6236a10e14"} Oct 30 21:43:05.025454 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.025430 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:43:05.025592 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.025508 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xc8xg\"" Oct 30 21:43:05.025592 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.025535 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:43:05.030040 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030017 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.030152 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030050 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-wtmp\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030152 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-tls\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030273 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.030273 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4nvs\" (UniqueName: \"kubernetes.io/projected/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-kube-api-access-p4nvs\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.030414 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.030414 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ct78\" (UniqueName: \"kubernetes.io/projected/3a0630d9-1d24-4533-ac24-c19bab683ae3-kube-api-access-6ct78\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030414 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-textfile\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030562 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030420 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-metrics-client-ca\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030562 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-sys\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030562 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030480 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-root\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030562 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.030562 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.030527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-accelerators-collector-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.031252 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.031214 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.038049 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.037855 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.040831 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.040809 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.052223 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.052197 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lbksg" event={"ID":"48ea24ed-9083-4214-ac43-06a49f4611dc","Type":"ContainerStarted","Data":"e1a6373f904248a60755f4fb2980a948ab745a239a2ae5d68c4f4fb4d539870f"} Oct 30 21:43:05.087396 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.087359 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4nvs\" (UniqueName: \"kubernetes.io/projected/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-kube-api-access-p4nvs\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.088761 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.088737 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:43:05.119788 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.119727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:05.131798 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131770 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-tls\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.131909 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ct78\" (UniqueName: \"kubernetes.io/projected/3a0630d9-1d24-4533-ac24-c19bab683ae3-kube-api-access-6ct78\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.131909 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-textfile\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132021 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-metrics-client-ca\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132021 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-sys\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132021 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-root\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132021 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.131999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132206 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.132025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-accelerators-collector-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132206 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.132092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-wtmp\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.132206 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.132184 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-root\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.133378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.132517 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-textfile\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.133378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.132595 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-sys\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.133378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.132662 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-wtmp\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.133378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.133038 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-accelerators-collector-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.133378 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.133242 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-metrics-client-ca\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.134524 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.134502 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-tls\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.135950 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.135911 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.187900 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.187862 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ct78\" (UniqueName: \"kubernetes.io/projected/3a0630d9-1d24-4533-ac24-c19bab683ae3-kube-api-access-6ct78\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.224200 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.224175 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:05.352395 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.352357 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:05.383019 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.382991 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:05.392968 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:43:05.392937 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0630d9_1d24_4533_ac24_c19bab683ae3.slice/crio-18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9 WatchSource:0}: Error finding container 18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9: Status 404 returned error can't find the container with id 18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9 Oct 30 21:43:05.612697 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.612634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:05.676331 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.676198 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lbksg" podStartSLOduration=35.232452479 podStartE2EDuration="38.676177549s" totalImagesPullingTime="3.44372507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:05.202841537 +0000 UTC m=+69.041723250" watchObservedRunningTime="2026-10-30 21:43:05.676177549 +0000 UTC m=+69.515059259" Oct 30 21:43:05.677298 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.677273 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6"] Oct 30 21:43:05.680275 ip-10-0-134-73 kubenswrapper[2570]: W1030 21:43:05.680240 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb804c3_d2fb_48bb_be14_3b1c6743f49f.slice/crio-e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac WatchSource:0}: Error finding container e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac: Status 404 returned error can't find the container with id e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac Oct 30 21:43:05.879231 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:05.879199 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:43:06.059962 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.059833 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qmc6j" event={"ID":"3a0630d9-1d24-4533-ac24-c19bab683ae3","Type":"ContainerStarted","Data":"18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9"} Oct 30 21:43:06.063583 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.063553 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" event={"ID":"ccb804c3-d2fb-48bb-be14-3b1c6743f49f","Type":"ContainerStarted","Data":"198c013f1913debe0fa0d63fa67d80305f2d5002ffcec18d555eab65296b0957"} Oct 30 21:43:06.063724 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.063611 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" event={"ID":"ccb804c3-d2fb-48bb-be14-3b1c6743f49f","Type":"ContainerStarted","Data":"86e04234f686c48cb377a85efdc7b0220d3fb31ad4896abd05606d506a4f3e59"} Oct 30 21:43:06.063724 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.063639 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" event={"ID":"ccb804c3-d2fb-48bb-be14-3b1c6743f49f","Type":"ContainerStarted","Data":"e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac"} Oct 30 21:43:06.065470 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.065446 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpfn7" event={"ID":"6b6edcdf-aa04-4afd-870c-4670f562dc06","Type":"ContainerStarted","Data":"97fcf601205737e6fd45b1e1def4300e242fc21f53a9fa16de26f0693ecfdbed"} Oct 30 21:43:06.313557 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.313485 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:06.477104 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.477037 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:06.780907 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:06.780872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:07.076867 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:07.076830 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-djc8v" event={"ID":"03cbfbcf-0c69-4649-a966-c29bb2ef0d05","Type":"ContainerStarted","Data":"507d5c4c8f789cc478fffe5eaea0140c3faee7964a935d5aacedf96a8c14ca4d"} Oct 30 21:43:07.077176 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:07.077150 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:07.093401 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:07.093352 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:07.190886 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:07.190831 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bpfn7" podStartSLOduration=67.684737615 podStartE2EDuration="1m10.19081614s" totalImagesPullingTime="2.506078525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:06.177219667 +0000 UTC m=+70.016101396" watchObservedRunningTime="2026-10-30 21:43:07.19081614 +0000 UTC m=+71.029697855" Oct 30 21:43:07.191148 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:07.191069 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-djc8v" podStartSLOduration=2.790012164 podStartE2EDuration="20.191062116s" totalImagesPullingTime="17.401049952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:07.190503705 +0000 UTC m=+71.029385526" watchObservedRunningTime="2026-10-30 21:43:07.191062116 +0000 UTC m=+71.029943828" Oct 30 21:43:07.284738 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:07.284708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:08.089066 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:08.089030 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a0630d9-1d24-4533-ac24-c19bab683ae3" containerID="03f1645c94edb5f9606131444df03618357cc2681d22b720e16049ac2b2a62ae" exitCode=0 Oct 30 21:43:08.089594 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:08.089123 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qmc6j" event={"ID":"3a0630d9-1d24-4533-ac24-c19bab683ae3","Type":"ContainerDied","Data":"03f1645c94edb5f9606131444df03618357cc2681d22b720e16049ac2b2a62ae"} Oct 30 21:43:09.096230 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.096190 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" event={"ID":"ccb804c3-d2fb-48bb-be14-3b1c6743f49f","Type":"ContainerStarted","Data":"6decedbc6644e0ee40b205033db8790a42f87441385cb18dfdfc2470fbe14f6e"} Oct 30 21:43:09.099148 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.099120 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qmc6j" event={"ID":"3a0630d9-1d24-4533-ac24-c19bab683ae3","Type":"ContainerStarted","Data":"9ae460dabca2d8af6396c98cc1a351817fea02af7bd287aa73fee1db14fd41c0"} Oct 30 21:43:09.099414 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.099177 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qmc6j" event={"ID":"3a0630d9-1d24-4533-ac24-c19bab683ae3","Type":"ContainerStarted","Data":"5e88cc65f5213ec391594d12d1a875031db13d9c4812f0d4506fd1f1b2568152"} Oct 30 21:43:09.198107 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.198064 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.333271 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.333198 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qmc6j" podStartSLOduration=3.420878512 podStartE2EDuration="5.333180291s" totalImagesPullingTime="1.801072406s" totalInitContainerRuntime="111.229373ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.332195245 +0000 UTC m=+73.171076956" watchObservedRunningTime="2026-10-30 21:43:09.333180291 +0000 UTC m=+73.172062002" Oct 30 21:43:09.334084 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.334042 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" podStartSLOduration=3.009889724 podStartE2EDuration="5.334025828s" totalImagesPullingTime="2.324136104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.216822376 +0000 UTC m=+73.055704087" watchObservedRunningTime="2026-10-30 21:43:09.334025828 +0000 UTC m=+73.172907541" Oct 30 21:43:09.362135 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.362107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.531952 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.531918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.702048 ip-10-0-134-73 kubenswrapper[2570]: I1030 21:43:09.701968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:09.717466 ip-10-0-134-73 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:43:09.729785 ip-10-0-134-73 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:43:09.729990 ip-10-0-134-73 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:43:09.730048 ip-10-0-134-73 systemd[1]: kubelet.service: Consumed 3.866s CPU time. Oct 30 21:43:09.748974 ip-10-0-134-73 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:43:09.848078 ip-10-0-134-73 kubenswrapper[5508]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:43:09.848078 ip-10-0-134-73 kubenswrapper[5508]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:43:09.848078 ip-10-0-134-73 kubenswrapper[5508]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:43:09.848078 ip-10-0-134-73 kubenswrapper[5508]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:43:09.850856 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850747 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.850856 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850853 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.850856 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850857 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.850856 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850861 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.850856 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850865 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850871 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850876 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850884 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850889 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850894 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850899 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850904 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850908 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850913 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850917 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850921 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850924 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850927 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850929 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850933 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850938 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850942 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850945 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.851120 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850949 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850954 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850958 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850969 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850972 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850976 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850978 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850982 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850986 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850989 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850992 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850995 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.850997 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851001 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851006 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851010 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851013 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851018 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851023 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851027 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.851852 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851032 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851035 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851040 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851043 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851045 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851048 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851052 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851055 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851058 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851061 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851065 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851070 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851074 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851079 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851083 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851087 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851092 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851096 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851099 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.852518 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851101 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851104 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851106 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851109 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851112 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851116 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851120 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851124 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851128 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851131 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851134 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851136 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851140 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851145 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851149 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851154 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851164 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851167 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851169 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851172 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.853193 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851176 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851180 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851185 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851190 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851193 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851195 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851198 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851202 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851205 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851208 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851210 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851213 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851216 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851219 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851221 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851227 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851232 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851243 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851246 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851250 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.853945 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851254 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851258 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851262 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.851267 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852386 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852398 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852402 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852405 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852407 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852416 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852419 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852422 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852425 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852428 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852431 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852435 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852440 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852444 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852448 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852457 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.854665 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852461 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852465 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852470 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852474 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852479 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852483 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852488 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852492 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852496 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852501 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852505 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852509 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852514 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852518 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852521 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852524 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852527 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852529 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852532 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.855466 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852535 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852538 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852542 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852546 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852549 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852551 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852556 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852560 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852565 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852570 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852574 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852579 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852590 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852594 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852598 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852600 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852603 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852606 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852609 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.856094 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852611 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852615 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852618 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852622 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852626 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852630 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852634 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852639 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852643 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852647 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852652 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852656 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852660 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852664 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852668 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852673 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852678 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852688 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852691 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.856814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852695 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852698 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852702 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852706 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852710 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852715 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852719 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852723 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852728 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852732 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852735 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852739 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852744 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852748 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852752 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852757 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852761 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852765 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852770 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852772 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.857522 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852775 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852777 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852779 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852782 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852784 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852787 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852790 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852792 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852797 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852801 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852808 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852825 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.852830 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853612 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853622 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853627 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853632 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853637 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853641 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.858144 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853645 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853648 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853651 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853653 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853656 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853659 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853664 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853668 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853673 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853678 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853683 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853688 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853693 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853697 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853701 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853705 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853709 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853714 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853718 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.858884 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853723 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853727 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853731 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853735 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853739 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853744 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853748 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853753 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853756 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853761 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853765 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853769 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853773 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853778 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853782 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853787 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853791 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853795 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853799 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853804 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.859574 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853808 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853812 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853818 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853825 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853832 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853836 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853841 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853846 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853851 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853855 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853859 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853864 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853869 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853873 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853878 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853882 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853886 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853890 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853895 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853900 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.860235 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853904 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853910 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853914 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853919 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853923 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853927 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853932 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853935 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853940 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853944 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853949 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853954 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853961 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853967 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853972 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853977 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853983 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853990 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853995 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.861021 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.853999 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854004 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854007 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854009 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854012 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854015 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854018 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854020 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854023 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854025 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854028 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854030 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854033 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854036 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854040 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854044 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854049 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854055 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854059 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.861861 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854061 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854063 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.854066 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854215 5508 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854225 5508 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854236 5508 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854241 5508 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854246 5508 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854249 5508 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854255 5508 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854264 5508 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854275 5508 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854280 5508 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854286 5508 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854291 5508 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854296 5508 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854316 5508 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854320 5508 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854323 5508 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854326 5508 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854329 5508 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854333 5508 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854336 5508 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:43:09.862722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854339 5508 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854342 5508 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854346 5508 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854351 5508 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854354 5508 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854357 5508 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854361 5508 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854372 5508 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854377 5508 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854382 5508 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854387 5508 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854391 5508 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854397 5508 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854402 5508 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854406 5508 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854411 5508 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854415 5508 flags.go:64] FLAG: --enable-server="true" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854420 5508 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854428 5508 flags.go:64] FLAG: --event-burst="100" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854434 5508 flags.go:64] FLAG: --event-qps="50" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854441 5508 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854445 5508 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854448 5508 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854453 5508 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:43:09.863736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854458 5508 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854464 5508 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854469 5508 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854474 5508 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854478 5508 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854481 5508 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854486 5508 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854491 5508 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854495 5508 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854500 5508 flags.go:64] FLAG: --feature-gates="" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854512 5508 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854517 5508 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854524 5508 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854530 5508 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854535 5508 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854541 5508 flags.go:64] FLAG: --help="false" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854546 5508 flags.go:64] FLAG: --hostname-override="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854551 5508 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854557 5508 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854562 5508 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854567 5508 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854574 5508 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854578 5508 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:43:09.864608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854583 5508 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854587 5508 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854591 5508 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854596 5508 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854606 5508 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854614 5508 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854619 5508 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854623 5508 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854629 5508 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854634 5508 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854639 5508 flags.go:64] FLAG: --lock-file="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854644 5508 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854649 5508 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854654 5508 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854661 5508 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854666 5508 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854671 5508 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854676 5508 flags.go:64] FLAG: --logging-format="text" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854682 5508 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854689 5508 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854692 5508 flags.go:64] FLAG: --manifest-url="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854695 5508 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854701 5508 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854704 5508 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854708 5508 flags.go:64] FLAG: --max-pods="110" Oct 30 21:43:09.865284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854711 5508 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854714 5508 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854717 5508 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854720 5508 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854723 5508 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854726 5508 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854730 5508 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854737 5508 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854740 5508 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854743 5508 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854746 5508 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854750 5508 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854754 5508 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854757 5508 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854760 5508 flags.go:64] FLAG: --port="10250" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854763 5508 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854766 5508 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e343f3552af8dc5b" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854771 5508 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854776 5508 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854782 5508 flags.go:64] FLAG: --register-node="true" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854786 5508 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854790 5508 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854795 5508 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854799 5508 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854804 5508 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854810 5508 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:43:09.866125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854815 5508 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854820 5508 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854825 5508 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854830 5508 flags.go:64] FLAG: --runonce="false" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854836 5508 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854841 5508 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854846 5508 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854850 5508 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854855 5508 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854860 5508 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854865 5508 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854871 5508 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854876 5508 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854881 5508 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854886 5508 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854892 5508 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854898 5508 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854904 5508 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854910 5508 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854919 5508 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854923 5508 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854929 5508 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854934 5508 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854936 5508 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854939 5508 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:43:09.866960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854942 5508 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854947 5508 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854952 5508 flags.go:64] FLAG: --v="2" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854959 5508 flags.go:64] FLAG: --version="false" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854966 5508 flags.go:64] FLAG: --vmodule="" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854972 5508 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.854978 5508 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855187 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855193 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855197 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855199 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855202 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855206 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855209 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855211 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855214 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855216 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855219 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855222 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855224 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855226 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.867870 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855229 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855231 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855234 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855238 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855242 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855244 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855247 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855252 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855256 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855260 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855265 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855268 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855271 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855273 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855276 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855279 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855281 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855284 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855286 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855290 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.868814 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855294 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855298 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855321 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855326 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855331 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855336 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855340 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855345 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855349 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855353 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855358 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855363 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855368 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855372 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855376 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855382 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855387 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855392 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855397 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855401 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.869669 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855406 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855410 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855415 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855419 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855424 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855428 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855433 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855436 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855439 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855441 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855446 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855450 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855453 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855456 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855458 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855461 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855463 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855467 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855469 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855472 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.870465 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855476 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855478 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855481 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855483 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855487 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855489 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855492 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855495 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855499 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855502 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855505 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855509 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855512 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855514 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855517 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855520 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855522 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855524 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855527 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.871086 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855529 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855532 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855534 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855537 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855539 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855542 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855544 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855547 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855551 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855555 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855560 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855565 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855570 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.855607 5508 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855650 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855656 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.871781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855661 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855665 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855670 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855674 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855679 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855684 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855688 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855693 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855700 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855707 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855713 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855718 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855723 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855728 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855732 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855735 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855738 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855740 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855743 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.872444 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855745 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855748 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855750 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855753 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855755 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855757 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855760 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855762 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855765 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855770 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855773 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855776 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855779 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855781 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855784 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855786 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855788 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855791 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855795 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.873177 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855798 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855800 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855802 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855806 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855811 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855815 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855820 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855823 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855825 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855828 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855830 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855833 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855836 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855838 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855841 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855843 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855846 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855848 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855851 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.873988 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855853 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855856 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855858 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855861 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855864 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855866 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855869 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855871 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855874 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855876 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855879 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855881 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855886 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855892 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855896 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855900 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855904 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855908 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855913 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855918 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.874848 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855921 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855924 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855926 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855929 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855931 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855934 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855937 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855939 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855942 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855944 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855947 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855949 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855952 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855954 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855957 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855959 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855961 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855965 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855969 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855974 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.875787 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855978 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855982 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855988 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855992 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.855997 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.856002 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.856005 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.861612 5508 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.861629 5508 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.861655 5508 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.861662 5508 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861917 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861926 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861931 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861936 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861940 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861945 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861949 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861954 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.876679 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861959 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861963 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861968 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861972 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861982 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861986 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861990 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861994 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.861999 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862003 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862026 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862031 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862035 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862039 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862044 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862049 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862053 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862062 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862067 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.877546 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862072 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862076 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862080 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862085 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862089 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862093 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862097 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862101 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862106 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862110 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862119 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862123 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862127 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862132 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862136 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862140 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862144 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862148 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862153 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862157 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.878386 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862162 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862166 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862170 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862179 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862202 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862207 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862211 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862216 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862221 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862226 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862230 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862235 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862239 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862243 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862247 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862258 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862263 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862268 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862272 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.879252 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862276 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862280 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862284 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862289 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862293 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862297 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862326 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862331 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862335 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862346 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862352 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862357 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862369 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862373 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862377 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862382 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862386 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862390 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862414 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862420 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.880140 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862424 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862433 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862437 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862441 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862446 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862450 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862455 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862459 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862463 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862468 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862472 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862476 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862481 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862485 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862493 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862498 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862502 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862506 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862510 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.881036 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.862514 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.862571 5508 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863362 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863380 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863386 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863391 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863396 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863400 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863403 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863410 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863416 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863420 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863423 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863427 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863430 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.882029 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863433 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863435 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863438 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863442 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863446 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863451 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863456 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863460 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863465 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863469 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863472 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863475 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863477 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863482 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863484 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863487 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863489 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863492 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863495 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.882830 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863499 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863504 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863508 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863513 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863517 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863522 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863526 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863530 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863534 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863539 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863543 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863548 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863551 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863554 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863557 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863560 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863563 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863566 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863569 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863572 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.883752 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863574 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863577 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863580 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863582 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863585 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863588 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863592 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863596 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863601 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863604 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863607 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863609 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863612 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863615 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863618 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863622 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863626 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863630 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863635 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863641 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.884677 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863645 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863652 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863659 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863663 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863666 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863669 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863672 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863675 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863677 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863681 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863686 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863690 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863694 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863698 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863703 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863707 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863712 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863717 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863723 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863727 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.885656 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863731 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863736 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863740 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863746 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863751 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863755 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863759 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863764 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863769 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863774 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863778 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863782 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863787 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863791 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.863829 5508 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863927 5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:43:09.886670 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863933 5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863936 5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863940 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863943 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863948 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863952 5508 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863956 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863961 5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863967 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863971 5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863975 5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863978 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863981 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863984 5508 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863987 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863990 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.863997 5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864003 5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864007 5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:43:09.887362 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864011 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864015 5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864021 5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864026 5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864030 5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864034 5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864036 5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864039 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864041 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864044 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864048 5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864052 5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864057 5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864060 5508 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864066 5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864070 5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864074 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864078 5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864082 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:43:09.887950 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864085 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864088 5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864091 5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864095 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864099 5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864103 5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864107 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864111 5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864115 5508 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864120 5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864124 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864128 5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864132 5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864135 5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864138 5508 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864140 5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864144 5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864148 5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864153 5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864156 5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:43:09.888551 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864159 5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864163 5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864167 5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864171 5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864176 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864179 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864183 5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864187 5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864193 5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864198 5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864203 5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864207 5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864212 5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864216 5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864221 5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864226 5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864230 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864234 5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864238 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:43:09.889248 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864243 5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864248 5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864252 5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864256 5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864259 5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864264 5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864267 5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864271 5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864273 5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864276 5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864279 5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864281 5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864284 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864286 5508 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864289 5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864291 5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864295 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864299 5508 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864319 5508 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864324 5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:43:09.889976 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864329 5508 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864332 5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864335 5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864338 5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864340 5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864344 5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864349 5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:09.864354 5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.864452 5508 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.865649 5508 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.865720 5508 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.865961 5508 server.go:1014] "Starting client certificate rotation" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.866109 5508 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.866163 5508 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:57 +0000 UTC" deadline="2028-06-12 17:39:28.942247239 +0000 UTC" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.866203 5508 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14179h56m19.076047638s" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.866598 5508 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:43:09.890725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.868006 5508 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:43:09.891371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.869670 5508 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:43:09.891371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.871791 5508 log.go:25] "Validated CRI v1 image API" Oct 30 21:43:09.891371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.872679 5508 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:43:09.891371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.875633 5508 fs.go:135] Filesystem UUIDs: map[159e24d9-df56-442d-bd8a-07b8d8815401:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dfb2a9ed-e6a8-49e9-8fd8-5348d3940aae:/dev/nvme0n1p4] Oct 30 21:43:09.891532 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.875661 5508 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03cbfbcf-0c69-4649-a966-c29bb2ef0d05/volumes/kubernetes.io~projected/kube-api-access-6m5f8:{mountpoint:/var/lib/kubelet/pods/03cbfbcf-0c69-4649-a966-c29bb2ef0d05/volumes/kubernetes.io~projected/kube-api-access-6m5f8 major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ecd634-c544-41e3-82bd-b5e36a5d57cd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/04ecd634-c544-41e3-82bd-b5e36a5d57cd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~projected/kube-api-access-6ct78:{mountpoint:/var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~projected/kube-api-access-6ct78 major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~projected/kube-api-access-rp7xz:{mountpoint:/var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~projected/kube-api-access-rp7xz major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48ea24ed-9083-4214-ac43-06a49f4611dc/volumes/kubernetes.io~projected/kube-api-access-dxq22:{mountpoint:/var/lib/kubelet/pods/48ea24ed-9083-4214-ac43-06a49f4611dc/volumes/kubernetes.io~projected/kube-api-access-dxq22 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48ea24ed-9083-4214-ac43-06a49f4611dc/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/48ea24ed-9083-4214-ac43-06a49f4611dc/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~projected/kube-api-access-8q9kb:{mountpoint:/var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~projected/kube-api-access-8q9kb major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e31250d-f97f-4b8b-9836-2f77a5deb9ce/volumes/kubernetes.io~projected/kube-api-access-pwvv6:{mountpoint:/var/lib/kubelet/pods/5e31250d-f97f-4b8b-9836-2f77a5deb9ce/volumes/kubernetes.io~projected/kube-api-access-pwvv6 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b6edcdf-aa04-4afd-870c-4670f562dc06/volumes/kubernetes.io~projected/kube-api-access-f749v:{mountpoint:/var/lib/kubelet/pods/6b6edcdf-aa04-4afd-870c-4670f562dc06/volumes/kubernetes.io~projected/kube-api-access-f749v major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b6edcdf-aa04-4afd-870c-4670f562dc06/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6b6edcdf-aa04-4afd-870c-4670f562dc06/volumes/kubernetes.io~secret/metrics-certs major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb39913-45bd-4e03-8a53-e5a269238d4e/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/6cb39913-45bd-4e03-8a53-e5a269238d4e/volumes/kubernetes.io~secret/tls-certificates major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volumes/kubernetes.io~projected/kube-api-access-vsnhk:{mountpoint:/var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volumes/kubernetes.io~projected/kube-api-access-vsnhk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/kube-api-access-hq2dz:{mountpoint:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/kube-api-access-hq2dz major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/registry-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bacb62d-563f-428d-9bf1-25b14d6e59c0/volumes/kubernetes.io~projected/kube-api-access-r959h:{mountpoint:/var/lib/kubelet/pods/8bacb62d-563f-428d-9bf1-25b14d6e59c0/volumes/kubernetes.io~projected/kube-api-access-r959h major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bacb62d-563f-428d-9bf1-25b14d6e59c0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8bacb62d-563f-428d-9bf1-25b14d6e59c0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc0c153-0125-47d4-bf2d-27668920eb0b/volumes/kubernetes.io~projected/kube-api-access-tcp62:{mountpoint:/var/lib/kubelet/pods/9bc0c153-0125-47d4-bf2d-27668920eb0b/volumes/kubernetes.io~projected/kube-api-access-tcp62 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2d2e278-5632-4d36-87d8-69023ae58aa0/volumes/kubernetes.io~projected/kube-api-access-rwm2b:{mountpoint:/var/lib/kubelet/pods/b2d2e278-5632-4d36-87d8-69023ae58aa0/volumes/kubernetes.io~projected/kube-api-access-rwm2b major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2d2e278-5632-4d36-87d8-69023ae58aa0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b2d2e278-5632-4d36-87d8-69023ae58aa0/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b59e3ce2-a114-414a-bfdd-23a58df02c91/volumes/kubernetes.io~projected/kube-api-access-6ht2x:{mountpoint:/var/lib/kubelet/pods/b59e3ce2-a114-414a-bfdd-23a58df02c91/volumes/kubernetes.io~projected/kube-api-access-6ht2x major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b59e3ce2-a114-414a-bfdd-23a58df02c91/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b59e3ce2-a114-414a-bfdd-23a58df02c91/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~projected/kube-api-access-p4nvs:{mountpoint:/var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~projected/kube-api-access-p4nvs major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d86f6ae9-bad4-41c8-be3c-61e4225a0501/volumes/kubernetes.io~projected/kube-api-access-qvcgb:{mountpoint:/var/lib/kubelet/pods/d86f6ae9-bad4-41c8-be3c-61e4225a0501/volumes/kubernetes.io~projected/kube-api-access-qvcgb major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb2cef57-0b45-422b-9390-ee103989900e/volumes/kubernetes.io~projected/kube-api-access-ft54x:{mountpoint:/var/lib/kubelet/pods/eb2cef57-0b45-422b-9390-ee103989900e/volumes/kubernetes.io~projected/kube-api-access-ft54x major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb80ad79-d9ca-41d8-a676-8844585591d4/volumes/kubernetes.io~projected/kube-api-access-s4zqr:{mountpoint:/var/lib/kubelet/pods/eb80ad79-d9ca-41d8-a676-8844585591d4/volumes/kubernetes.io~projected/kube-api-access-s4zqr major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b/volumes/kubernetes.io~projected/kube-api-access-w6z2d:{mountpoint:/var/lib/kubelet/pods/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b/volumes/kubernetes.io~projected/kube-api-access-w6z2d major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3573ec-d74e-44d9-aec5-ca58f51e04fd/volumes/kubernetes.io~projected/kube-api-access-fm8vd:{mountpoint:/var/lib/kubelet/pods/fe3573ec-d74e-44d9-aec5-ca58f51e04fd/volumes/kubernetes.io~projected/kube-api-access-fm8vd major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/288cd6bb7248ec844204d41f1ed56d0b3cad33207fb532a16d5df580b7c2d729/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b0a82b3f82f44ee5086e1222a27a517be46a406eca5b10282bfc47d5103a07f8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7557a3dac2daa843a765698d8bf62891f71a1c47b8fb7f7570844eea9c38108c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2161e97bd499d1237f812671f0a8a33fceb3687259dfd7b9da1228dd349493a9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/9e5d144917c5aec5f61fc7cad63c26a4fc0e66f492f61f1d9e1d87b582f01a04/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4e59d5a5ae9056f287dbb3939565dd59af1ede0cf95860cf6c2d7ff84bc40d46/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/34e1897d08773a3c48b9681955d37fa10ba8471621a262bb88dddad91b551261/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a0bc8866dd60dda91b2037c6d4300c175d8113bb88ffd16899fb143b33e209e5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/cf373b2b13b22e1c7777f0f49ebee82b05426af566d8b80bb808f0cd88aa6ff9/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/15a67b47c30466357c499a98d43905dc54a941b6a6088b67e6a7613500113cb1/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/53e4fbcc5d0c1977078b9ae4c67e26e6ebad5a1c629048fb1e10541bdc634733/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/0f5dda5b03c463803afc44cbd5435139d4c40384947a8dff7131223a093e2ad1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fb1b324a7ce0ee6d16e263856ba12e6cf8b5ba2c4999cca076059eec4f161501/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/eed9aec846127df97b935f176dff47b43cd47ee9905f2b432347809684a9e27c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4e502f2be1ad24fd0f5da7025c6db33ed4d5d29bdd198ad2d12b1cf8fdfa398a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7844fdf527408264d4fed392e5d13983cbe36898a20c3bf644bfbf76bce8934e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/67cc32fbbe1db0d1e19b81faf4733c4653b88a4dd6bc8c7f2e6a818d546744ea/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/5750928001dfe2eb008a3169e98418c8d1a883626beb85e3e5c66d0928cfab88/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/b92d5d68193a07c1fc1feb72151876de450f267d4ddeb61ba4777a72d0bcac87/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-180:{mountpoint:/var/lib/containers/storage/overlay/e9f33d6a8af76cb607ee8fc1d6129ef02ce84334482813de376e9b5f4baa97eb/merged major:0 minor:180 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e70525bf45479e2cb2c490e40e61979209e696cf5dc86993ab0d255c11f71cfe/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/548d72ab4ccf9b410992a0c36e97556ca60d1b27190065d7d24dc19496adbe7d/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f3b3ef320da2322dca383a3370f3b9a52e78001f713436503034389a2b332c16/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/c8fff1fe07b861adc178cd4cc1faaa627d6365f584844bb03db4e8b1610c4263/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/a07055fef04eb75c1791b43e9a0db3a9930a3ff1dbec60b30cb8c6a747c4aa0a/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/c5647de62fa94ced2f138ce97338f34cd25ed89dfcec4bad48bcf41092492eab/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/ffdaa87539030ac42dbb797a149ff21e1b4dea1c9693427839586914a1789682/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/5f16793746fde92f689c5ef4bbf2d083604d37835936d6d298443e618c88001f/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/46e7eafbc000e64a82a5c91ae4213d5490a2e26790d9bb0d07e80bd0eb79674f/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/eec59767f61cb3c6a9f2794a871871070177ca924ece57cb2342d1ad400ca2fd/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/4fe1a1441d34a6b7c9feca2e5bdb27d5de47fd27ac7613c05d27166c38246b3b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/3a967466c4e62e0e399a858cdb3a5566b4cc182eeb35400d71e22f74097ce65a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/615dd099ec711d358f9774a94fe9cf885dd32301a358413281f023e47e9ddce7/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/fb93819e9f164de3bee4d68167a0a8b2556adc1e125c5c1c89ecfbd108a4e3a9/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/e81dcdb87c82a2b892e95a1a4dc1d61fefe42e16dd6b1762e9f94e0898edb7bb/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/47280eb91eab3cf6f5d80508a9b72eb3ae11fcc71b59fcc1c04f0005a731803b/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/f06c660e28820353d6587ecfc6eee2ebaffee6a3a5f99aa9204be7e66fe13d15/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/123ccafd8f9b960b19b71a4a58915838df0b90163d4073f5f1bf5fa99a429d9c/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/e7005444e5eb6e2fd76581c902a98ed181b5a4a25deb25d30bf2023da7b48c72/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/9e97a7f1b367d68dcd035f9c6820af3ef60c22a346590ddf19fa63700a8734a4/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/19a865f86ad627eeb0e4499bbe037bdcc58e7466695bb6950ec5e206f589ce69/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/79c1595cce46045bfb4722a0cc3f167d08fa725d187926a482394718ddab3d74/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/0a42b0a0d210cec9fe837b2c10a66cb12a4430e35c2a875d8f0a0f963379864c/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/b969c544907d6f5f673f1081ac3ff3bd1f197d756e6601effd62a1911d63e5c0/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/1c3d22a0c46c38fe3d84ce6f56250665614d09256da35a090d3e6e2f5631c0d1/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/abfa26aa6b2ff9a39e536e0864a23870cf883fee04dde5b3681a7eac27a20096/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/5ea7166621ecbb5caa060a48849513a4d19fc1c2719b73ead9de7ad9cec39e93/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/74d076ffa5363c48ffe0bdc77bee3cf94f61a46c301bc2ed9aca0c18a21078c2/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/30e2b3eb5b4bbc04f6ff02589ba390929bd13136e65a0a4f74b57d26d1645774/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/f82cdc7157172ae107be8ed8be763aa343bfa1e4ff5e1612db16d70d409ad002/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/b726d7ab3b94a264eee8f3157bfdcd657175ba6e9fed7d718d4fe8fbac66929c/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/6621d6b2ea7e1a8d986189288290630dfb9bd83d5dc3d7b173f5754cc1493f35/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/53227a64e9d560067142deb74b0216487def271826b860b97642530040ea6acc/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/70a36ed18569cdf8efdc18f4483a85d7067f0c9e78bc07b5821239923665ef15/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/c5a1a84e3856bd90be274e375c8e2f2bb70b46760e97c0c6a3d57328dc379ea9/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/94ca80afdc85f4b6f1909fd943256988c4a2c57f25bc9d6a3a09b0c7ae8556da/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/2476fabf7fb52f7fc13db28748ef27a040ad2aea07db091c82f97397c3cacb6f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a0467591cd25b8837ee7f8d5e50c86814107c574e53017b58e993bd29842765f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/bf545511c51c6443fe393b11f81e305db8b77aab1ca322ba556ca3aa6bba78ad/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a99394b6e46c4437f306bd851d8325828a7b5bdf824dd3b744d56b950ee904f3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/77d4f3645b8a709fe7e1593fb519e3e28dbda3bb65d63a15d9b10d53a489db98/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/220b3eebc2d610f05de0d0ddad10b0e28b3e82c18a078536641dcc4505a755c2/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8cd8a6fc1a8c1e168618d07ff22f281e7531571347a58cd340c60816550ce3ad/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6682d6d430c4484e3aa2250718bc86f9a89a930ded6be48649af75ccc4192bfe/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/d687fb4287025a623f66841572ae72615204d228b372ab532ae97fa6f651befb/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.888141 5508 manager.go:217] Machine: {Timestamp:2026-10-30 21:43:09.887399435 +0000 UTC m=+0.076937106 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099441 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec243e6063db9c2b43a85d9deacaaa69 SystemUUID:ec243e60-63db-9c2b-43a8-5d9deacaaa69 BootID:1529fa24-5dee-4742-ab76-4e180d3b1843 Filesystems:[{Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b749ae4f1f70132a44e8b96421d6304ef6e1afa880a73149734e01f7f1c9f02/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e31250d-f97f-4b8b-9836-2f77a5deb9ce/volumes/kubernetes.io~projected/kube-api-access-pwvv6 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb80ad79-d9ca-41d8-a676-8844585591d4/volumes/kubernetes.io~projected/kube-api-access-s4zqr DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volumes/kubernetes.io~projected/kube-api-access-vsnhk DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3573ec-d74e-44d9-aec5-ca58f51e04fd/volumes/kubernetes.io~projected/kube-api-access-fm8vd DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92c1035e54db8106c410a264a305d412251f236567299a369af3868e48bda908/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b/volumes/kubernetes.io~projected/kube-api-access-w6z2d DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7787a55b24246bc6fdcff3f880a348e3a6b2971c3349d09e51043529ec529f6a/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/48ea24ed-9083-4214-ac43-06a49f4611dc/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f309e472873b7ac1d1d0f95ccd72716bbffff757855d3723cee38aa634d3cfd/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb39913-45bd-4e03-8a53-e5a269238d4e/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:268 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9db336c7e3d6997e250b25a8e4754ca822101dba2964073ff858efed6c4c14cf/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48ea24ed-9083-4214-ac43-06a49f4611dc/volumes/kubernetes.io~projected/kube-api-access-dxq22 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b6edcdf-aa04-4afd-870c-4670f562dc06/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d7cf8bbc8b751bedf9d6cc2d5c00df151bc02e984f3e9a2d7ac4ceace90f1e4/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8bacb62d-563f-428d-9bf1-25b14d6e59c0/volumes/kubernetes.io~projected/kube-api-access-r959h DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81f74677c4a7f5cdbeb4ab956957f51126f8b0db1c6980a860f152de2c84c133/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3c6b3f70b1230cdbbde17ade114ed8c3a808e5f584e7f1aa109cb97e40065ac/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~projected/kube-api-access-rp7xz DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b6edcdf-aa04-4afd-870c-4670f562dc06/volumes/kubernetes.io~projected/kube-api-access-f749v DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f58ff13468228f09ccb942fc2c812432552cf48578b7f6cf2b1d46dd4b20fb10/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/03cbfbcf-0c69-4649-a966-c29bb2ef0d05/volumes/kubernetes.io~projected/kube-api-access-6m5f8 DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd401fe9c8b4e13a092aba88f05fd22a8d91c7f25c27d017080271c846a322c2/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/442ddd933de0017c1126a835d13c84732c082c1c41d83f6ecbc6acd4bb9fb0dd/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d31591ccfcd769b573f9d38c1ceb72270971f16bc30272b96f3c0fb768ce6362/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e7a5e7ad9e1fab0fa563e42af4cea7e8201beae2d10313a83d450f351fb17e3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:255 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~projected/kube-api-access-p4nvs DeviceMajor:0 DeviceMinor:345 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18bd704ab6a7a95160031c3276fb653cc8f9847f0cb7e94c5778179823730ed9/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41738995-836d-4314-baa7-d2e7f5149a56/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:249 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc0c153-0125-47d4-bf2d-27668920eb0b/volumes/kubernetes.io~projected/kube-api-access-tcp62 DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b59e3ce2-a114-414a-bfdd-23a58df02c91/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70d8105d-db42-4ae1-9265-028ab1488148/volumes/kubernetes.io~projected/kube-api-access-hq2dz DeviceMajor:0 DeviceMinor:256 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/50ebddb7-2e87-4a26-844e-bcb12013c7c2/volumes/kubernetes.io~projected/kube-api-access-8q9kb DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04ecd634-c544-41e3-82bd-b5e36a5d57cd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bacb62d-563f-428d-9bf1-25b14d6e59c0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42f9b3aafa69be1b216b277ce7959732c56f724d3985b8ac2e22eade1eadee67/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccb804c3-d2fb-48bb-be14-3b1c6743f49f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eb2cef57-0b45-422b-9390-ee103989900e/volumes/kubernetes.io~projected/kube-api-access-ft54x DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/980e19be85d1a5ca2150a386d568f5ab9102e6638f649683d6ff6189c4bd7555/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-180 DeviceMajor:0 DeviceMinor:180 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2d2e278-5632-4d36-87d8-69023ae58aa0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4530df5654a3119938bf3286e4b8f8c7b025f695d34a0a36475a81dbc6e1c0db/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d86f6ae9-bad4-41c8-be3c-61e4225a0501/volumes/kubernetes.io~projected/kube-api-access-qvcgb DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ee2026a91bcfa365342235f0f185ab5f48d2e43d0dfcd989786761b62c13241/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24c7b95ad066c74e51136907e7dc359f2f0e2426c2251f12dcd28fe11dfb236c/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5236bf68c8e56a4113ee203c88339c5a8bad7b5bb51a431811520ae3ac11116/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b59e3ce2-a114-414a-bfdd-23a58df02c91/volumes/kubernetes.io~projected/kube-api-access-6ht2x DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13196f515f6b4411367fe8514ea6338f05349640ed42560295fb1c537b8bcbd2/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3a0630d9-1d24-4533-ac24-c19bab683ae3/volumes/kubernetes.io~projected/kube-api-access-6ct78 DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2d2e278-5632-4d36-87d8-69023ae58aa0/volumes/kubernetes.io~projected/kube-api-access-rwm2b DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:13196f515f6b441 MacAddress:52:93:c7:e4:61:23 Speed:10000 Mtu:8901} {Name:24c7b95ad066c74 MacAddress:be:2b:dd:1f:55:c8 Speed:10000 Mtu:8901} {Name:442ddd933de0017 MacAddress:5a:fc:28:bc:7e:29 Speed:10000 Mtu:8901} {Name:4530df5654a3119 MacAddress:9a:74:88:ba:ef:39 Speed:10000 Mtu:8901} {Name:4b749ae4f1f7013 MacAddress:b2:c6:ee:52:ab:a2 Speed:10000 Mtu:8901} {Name:7787a55b24246bc MacAddress:b6:82:eb:40:12:82 Speed:10000 Mtu:8901} {Name:81f74677c4a7f5c MacAddress:86:13:e4:d3:6e:56 Speed:10000 Mtu:8901} {Name:8f309e472873b7a MacAddress:1a:7c:d9:15:f4:7a Speed:10000 Mtu:8901} {Name:9db336c7e3d6997 MacAddress:82:39:eb:c0:ae:7e Speed:10000 Mtu:8901} {Name:bd401fe9c8b4e13 MacAddress:3e:ae:aa:80:37:15 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:5d:c7:2e:de:e3 Speed:0 Mtu:9001} {Name:br-int MacAddress:52:26:a4:4c:87:0e Speed:0 Mtu:8901} {Name:e3c6b3f70b1230c MacAddress:c2:aa:78:95:82:ff Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:5d:c7:2e:de:e3 Speed:0 Mtu:9001} {Name:f58ff13468228f0 MacAddress:da:90:1a:31:da:f6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1e:86:21:77:b4:7f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:2b:8c:27:06:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.888674 5508 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.888732 5508 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889180 5508 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889218 5508 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-73.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889391 5508 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889399 5508 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889411 5508 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889433 5508 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889528 5508 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889742 5508 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889890 5508 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889930 5508 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889940 5508 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889949 5508 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889959 5508 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.889966 5508 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.890915 5508 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.890928 5508 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:43:09.891882 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.891477 5508 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:43:09.894706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.894685 5508 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:43:09.894706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.894704 5508 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:43:09.894872 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.894835 5508 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:43:09.895058 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895043 5508 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:43:09.895112 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895085 5508 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:43:09.895257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895230 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:43:09.895257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895258 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895268 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895278 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895286 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895295 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895327 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895336 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895343 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895349 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895357 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895367 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895402 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:43:09.895440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.895408 5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:43:09.899512 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.899494 5508 server.go:1263] "Started kubelet" Oct 30 21:43:09.900902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.899815 5508 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:43:09.900902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.899930 5508 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:43:09.900117 ip-10-0-134-73 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:43:09.901472 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.901181 5508 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:43:09.901829 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.901607 5508 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:43:09.909576 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.908474 5508 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:43:09.911236 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.911218 5508 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.911724 5508 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.912067 5508 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.912078 5508 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:09.912360 5508 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.912703 5508 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.914088 5508 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:58 +0000 UTC" deadline="2028-08-13 11:42:27.181566436 +0000 UTC" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.914127 5508 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15661h59m17.267444144s" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.915894 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0630d9-1d24-4533-ac24-c19bab683ae3" volumeName="kubernetes.io/empty-dir/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.915948 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41738995-836d-4314-baa7-d2e7f5149a56" volumeName="kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-etc-tuned" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.915970 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ebddb7-2e87-4a26-844e-bcb12013c7c2" volumeName="kubernetes.io/configmap/50ebddb7-2e87-4a26-844e-bcb12013c7c2-metrics-client-ca" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.915984 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6edcdf-aa04-4afd-870c-4670f562dc06" volumeName="kubernetes.io/projected/6b6edcdf-aa04-4afd-870c-4670f562dc06-kube-api-access-f749v" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.915997 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ebddb7-2e87-4a26-844e-bcb12013c7c2" volumeName="kubernetes.io/projected/50ebddb7-2e87-4a26-844e-bcb12013c7c2-kube-api-access-8q9kb" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916014 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e" volumeName="kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-env-overrides" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916025 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccb804c3-d2fb-48bb-be14-3b1c6743f49f" volumeName="kubernetes.io/configmap/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-metrics-client-ca" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916055 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0630d9-1d24-4533-ac24-c19bab683ae3" volumeName="kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-metrics-client-ca" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916073 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" volumeName="kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916090 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" volumeName="kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-binary-copy" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916102 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0630d9-1d24-4533-ac24-c19bab683ae3" volumeName="kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916115 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e" volumeName="kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916133 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916144 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b59e3ce2-a114-414a-bfdd-23a58df02c91" volumeName="kubernetes.io/configmap/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-cabundle" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916166 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b59e3ce2-a114-414a-bfdd-23a58df02c91" volumeName="kubernetes.io/secret/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-key" seLinuxMountContext="" Oct 30 21:43:09.917233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.916179 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86f6ae9-bad4-41c8-be3c-61e4225a0501" volumeName="kubernetes.io/configmap/d86f6ae9-bad4-41c8-be3c-61e4225a0501-serviceca" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919399 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cbfbcf-0c69-4649-a966-c29bb2ef0d05" volumeName="kubernetes.io/projected/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-kube-api-access-6m5f8" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919435 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e" volumeName="kubernetes.io/projected/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-kube-api-access-vsnhk" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919449 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bacb62d-563f-428d-9bf1-25b14d6e59c0" volumeName="kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919464 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48ea24ed-9083-4214-ac43-06a49f4611dc" volumeName="kubernetes.io/empty-dir/48ea24ed-9083-4214-ac43-06a49f4611dc-data-volume" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919491 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e" volumeName="kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-config" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919504 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-registry-tls" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919516 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-kube-api-access-hq2dz" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.918556 5508 factory.go:55] Registering systemd factory Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919528 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc0c153-0125-47d4-bf2d-27668920eb0b" volumeName="kubernetes.io/projected/9bc0c153-0125-47d4-bf2d-27668920eb0b-kube-api-access-tcp62" seLinuxMountContext="" Oct 30 21:43:09.919562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919543 5508 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919542 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b59e3ce2-a114-414a-bfdd-23a58df02c91" volumeName="kubernetes.io/projected/b59e3ce2-a114-414a-bfdd-23a58df02c91-kube-api-access-6ht2x" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919619 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb80ad79-d9ca-41d8-a676-8844585591d4" volumeName="kubernetes.io/projected/eb80ad79-d9ca-41d8-a676-8844585591d4-kube-api-access-s4zqr" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919951 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc" volumeName="kubernetes.io/secret/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-agent-certs" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919977 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0630d9-1d24-4533-ac24-c19bab683ae3" volumeName="kubernetes.io/projected/3a0630d9-1d24-4533-ac24-c19bab683ae3-kube-api-access-6ct78" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.919995 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41738995-836d-4314-baa7-d2e7f5149a56" volumeName="kubernetes.io/projected/41738995-836d-4314-baa7-d2e7f5149a56-kube-api-access-rp7xz" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920012 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ebddb7-2e87-4a26-844e-bcb12013c7c2" volumeName="kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920023 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccb804c3-d2fb-48bb-be14-3b1c6743f49f" volumeName="kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920050 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb2cef57-0b45-422b-9390-ee103989900e" volumeName="kubernetes.io/projected/eb2cef57-0b45-422b-9390-ee103989900e-kube-api-access-ft54x" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920062 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0630d9-1d24-4533-ac24-c19bab683ae3" volumeName="kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920074 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e" volumeName="kubernetes.io/secret/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920086 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a0630d9-1d24-4533-ac24-c19bab683ae3" volumeName="kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-tls" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920102 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48ea24ed-9083-4214-ac43-06a49f4611dc" volumeName="kubernetes.io/projected/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-api-access-dxq22" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920113 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb39913-45bd-4e03-8a53-e5a269238d4e" volumeName="kubernetes.io/secret/6cb39913-45bd-4e03-8a53-e5a269238d4e-tls-certificates" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920125 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/empty-dir/70d8105d-db42-4ae1-9265-028ab1488148-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920136 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-bound-sa-token" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920148 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc0c153-0125-47d4-bf2d-27668920eb0b" volumeName="kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-cni-binary-copy" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920160 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86f6ae9-bad4-41c8-be3c-61e4225a0501" volumeName="kubernetes.io/projected/d86f6ae9-bad4-41c8-be3c-61e4225a0501-kube-api-access-qvcgb" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920175 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41738995-836d-4314-baa7-d2e7f5149a56" volumeName="kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-tmp" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920187 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-registry-certificates" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920199 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-trusted-ca" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920210 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b" volumeName="kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920223 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc0c153-0125-47d4-bf2d-27668920eb0b" volumeName="kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-daemon-config" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920236 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ecd634-c544-41e3-82bd-b5e36a5d57cd" volumeName="kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920250 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70d8105d-db42-4ae1-9265-028ab1488148" volumeName="kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920268 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bacb62d-563f-428d-9bf1-25b14d6e59c0" volumeName="kubernetes.io/configmap/8bacb62d-563f-428d-9bf1-25b14d6e59c0-config-volume" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920279 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" volumeName="kubernetes.io/projected/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-kube-api-access-pwvv6" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920291 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d2e278-5632-4d36-87d8-69023ae58aa0" volumeName="kubernetes.io/projected/b2d2e278-5632-4d36-87d8-69023ae58aa0-kube-api-access-rwm2b" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920321 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2d2e278-5632-4d36-87d8-69023ae58aa0" volumeName="kubernetes.io/secret/b2d2e278-5632-4d36-87d8-69023ae58aa0-cert" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920342 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccb804c3-d2fb-48bb-be14-3b1c6743f49f" volumeName="kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920355 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb2cef57-0b45-422b-9390-ee103989900e" volumeName="kubernetes.io/empty-dir/eb2cef57-0b45-422b-9390-ee103989900e-tmp-dir" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920366 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb80ad79-d9ca-41d8-a676-8844585591d4" volumeName="kubernetes.io/configmap/eb80ad79-d9ca-41d8-a676-8844585591d4-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920379 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3573ec-d74e-44d9-aec5-ca58f51e04fd" volumeName="kubernetes.io/projected/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kube-api-access-fm8vd" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920404 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6edcdf-aa04-4afd-870c-4670f562dc06" volumeName="kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920417 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48ea24ed-9083-4214-ac43-06a49f4611dc" volumeName="kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920429 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50ebddb7-2e87-4a26-844e-bcb12013c7c2" volumeName="kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-tls" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920443 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" volumeName="kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920460 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bacb62d-563f-428d-9bf1-25b14d6e59c0" volumeName="kubernetes.io/projected/8bacb62d-563f-428d-9bf1-25b14d6e59c0-kube-api-access-r959h" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920473 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cbfbcf-0c69-4649-a966-c29bb2ef0d05" volumeName="kubernetes.io/empty-dir/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-tmp" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920487 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc" volumeName="kubernetes.io/configmap/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-konnectivity-ca" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920501 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48ea24ed-9083-4214-ac43-06a49f4611dc" volumeName="kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920514 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bacb62d-563f-428d-9bf1-25b14d6e59c0" volumeName="kubernetes.io/empty-dir/8bacb62d-563f-428d-9bf1-25b14d6e59c0-tmp-dir" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920527 5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccb804c3-d2fb-48bb-be14-3b1c6743f49f" volumeName="kubernetes.io/projected/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-kube-api-access-p4nvs" seLinuxMountContext="" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920541 5508 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:43:09.920672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.920553 5508 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:43:09.926886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.921154 5508 factory.go:153] Registering CRI-O factory Oct 30 21:43:09.926886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.921172 5508 factory.go:223] Registration of the crio container factory successfully Oct 30 21:43:09.926886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.921229 5508 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:43:09.926886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.921249 5508 factory.go:103] Registering Raw factory Oct 30 21:43:09.926886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.921265 5508 manager.go:1199] Started watching for new ooms in manager Oct 30 21:43:09.926886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.921700 5508 manager.go:319] Starting recovery of all containers Oct 30 21:43:09.941785 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.941746 5508 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:43:09.944490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.944213 5508 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:43:09.944490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.944273 5508 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:43:09.944490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:09.944406 5508 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:43:09.944490 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:09.944450 5508 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:43:10.013440 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.012982 5508 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" Oct 30 21:43:10.017723 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.017637 5508 generic.go:356] "Generic (PLEG): container finished" podID="3a0630d9-1d24-4533-ac24-c19bab683ae3" containerID="03f1645c94edb5f9606131444df03618357cc2681d22b720e16049ac2b2a62ae" exitCode=0 Oct 30 21:43:10.032383 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.032355 5508 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="531015aa0c422875110bad7b288888d082b648ca6510b1c8df537c017944c4bc" exitCode=0 Oct 30 21:43:10.032383 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.032387 5508 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="c6c2ff618e38dd3a08c917a9d1c00612cd3d62548a0fe7d026021c90267f90ff" exitCode=0 Oct 30 21:43:10.032799 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.032395 5508 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="986042df4725cbc0c8608978d34b3b959d1db52e46cc57ef964515035337e34e" exitCode=0 Oct 30 21:43:10.032799 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.032402 5508 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="913c6b324709b148f01a2f910cb107059a9b04c0959ade9e35639a35774a0bc6" exitCode=0 Oct 30 21:43:10.032799 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.032407 5508 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="c030c8fb59fe1748fbd637c1750eacfbcf7316e0c524f53546b02140430f1811" exitCode=0 Oct 30 21:43:10.032799 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.032413 5508 generic.go:356] "Generic (PLEG): container finished" podID="5e31250d-f97f-4b8b-9836-2f77a5deb9ce" containerID="78d5e275c4b1c88461abba80394b2908591ff1f0790e436a42aa354e95017ffc" exitCode=0 Oct 30 21:43:10.039542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.039519 5508 generic.go:356] "Generic (PLEG): container finished" podID="99a99366fa1c27f76a4200901a9260b1" containerID="a794db469e508564fd6f94755b7e5c726affd9d03e242fb2fe8929aecfdd78e2" exitCode=0 Oct 30 21:43:10.045042 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.044825 5508 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:43:10.061672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.061650 5508 manager.go:324] Recovery completed Oct 30 21:43:10.084146 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.083741 5508 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:10.086112 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.086079 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:10.086212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.086121 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:10.086212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.086136 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:10.089274 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089251 5508 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:43:10.089274 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089270 5508 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:43:10.089450 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089292 5508 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:43:10.089450 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089259 5508 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:10.089545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089468 5508 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:43:10.089545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089484 5508 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:43:10.089545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089507 5508 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:43:10.089545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089515 5508 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:43:10.089545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089523 5508 policy_none.go:50] "Start" Oct 30 21:43:10.089757 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089550 5508 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:43:10.089757 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089565 5508 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:43:10.089757 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089727 5508 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:43:10.089757 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089734 5508 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:43:10.089757 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.089741 5508 policy_none.go:44] "Start" Oct 30 21:43:10.090375 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.090349 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:10.090479 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.090394 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:10.090479 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.090406 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:10.095194 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.095167 5508 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:43:10.098495 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.098478 5508 manager.go:341] "Starting Device Plugin manager" Oct 30 21:43:10.098588 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.098505 5508 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:43:10.098588 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.098514 5508 server.go:87] "Starting device plugin registration server" Oct 30 21:43:10.098826 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.098809 5508 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:43:10.098977 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.098823 5508 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:43:10.099089 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.099016 5508 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:43:10.099155 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.099104 5508 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:43:10.099155 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.099141 5508 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:43:10.099155 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.099148 5508 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:43:10.099485 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.099220 5508 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:43:10.099130692Z","UUID":"e2b5889a-e762-4e12-b198-1139239a8401","Handler":null,"Name":"","Endpoint":""} Oct 30 21:43:10.102051 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.102027 5508 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:43:10.102160 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.102071 5508 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-73.us-west-1.compute.internal\" not found" Oct 30 21:43:10.102207 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.102183 5508 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:43:10.102252 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.102209 5508 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:43:10.104425 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.104400 5508 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:10.104529 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.104438 5508 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:10.104722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.104705 5508 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:43:10.151627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.151600 5508 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:10.199630 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.199599 5508 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:43:10.200668 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.200644 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:43:10.200777 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.200683 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:43:10.200777 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.200698 5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-73.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:43:10.200777 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.200735 5508 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.201446 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.201420 5508 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:43:10.244953 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.244890 5508 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal"] Oct 30 21:43:10.245742 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.245699 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"0994e35dd707326725d576ce662475d4","Type":"ContainerStarted","Data":"4e59b3b4c41c22003c4febff248e308ca9626961ce2a80a8d4081868b078f88e"} Oct 30 21:43:10.245870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.245753 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"0994e35dd707326725d576ce662475d4","Type":"ContainerStarted","Data":"84799fddd185eb55d5eaf277f325d86557cc335814e66949776df010ddd6e9f1"} Oct 30 21:43:10.245870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.245834 5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.245870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.245841 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"99a99366fa1c27f76a4200901a9260b1","Type":"ContainerStarted","Data":"874bdd3c96c25bb9bfca0c2410fa3eeae3dedb3f46adb5e760f3a45d1acf046a"} Oct 30 21:43:10.245870 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.245859 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"99a99366fa1c27f76a4200901a9260b1","Type":"ContainerDied","Data":"a794db469e508564fd6f94755b7e5c726affd9d03e242fb2fe8929aecfdd78e2"} Oct 30 21:43:10.246034 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.245878 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" event={"ID":"99a99366fa1c27f76a4200901a9260b1","Type":"ContainerStarted","Data":"ce72f38aad805499824cf4f6c30a4444ee05ec0022a29702d885a80ef5565ba0"} Oct 30 21:43:10.246108 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.246093 5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.311743 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.311689 5508 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.311835 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.311746 5508 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.312339 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.312296 5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:43:10.312513 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.312491 5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.312608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.312351 5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:43:10.312608 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:10.312566 5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.406286 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.406257 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.406465 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.406292 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.406465 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.406337 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0994e35dd707326725d576ce662475d4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"0994e35dd707326725d576ce662475d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.507136 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.507039 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.507290 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.507169 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.507290 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.507104 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.507290 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.507234 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0994e35dd707326725d576ce662475d4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"0994e35dd707326725d576ce662475d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.507290 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.507199 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0994e35dd707326725d576ce662475d4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"0994e35dd707326725d576ce662475d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.507290 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.507246 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99a99366fa1c27f76a4200901a9260b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" (UID: \"99a99366fa1c27f76a4200901a9260b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:10.891828 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:10.891789 5508 apiserver.go:52] "Watching apiserver" Oct 30 21:43:11.044470 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.044435 5508 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:43:11.045138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.045112 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-8hwrm","openshift-image-registry/image-registry-5f7d486fcd-vl4dg","openshift-insights/insights-runtime-extractor-lbksg","openshift-multus/multus-qg64f","openshift-dns/node-resolver-tsn5z","openshift-ingress-canary/ingress-canary-jfjmh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct","openshift-monitoring/telemeter-client-5659774bc7-pwf5k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal","kube-system/konnectivity-agent-cggst","openshift-dns/dns-default-nwqgt","openshift-image-registry/node-ca-2r78l","openshift-network-diagnostics/network-check-target-ng9zm","kube-system/kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal","openshift-monitoring/metrics-server-6499d46c6f-54lvw","openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm","openshift-network-operator/iptables-alerter-5kndd","kube-system/global-pull-secret-syncer-56vhn","openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6","openshift-ovn-kubernetes/ovnkube-node-7mj55","openshift-service-ca/service-ca-5fffc97bdf-7qf6l","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5","openshift-multus/network-metrics-daemon-bpfn7","openshift-cluster-node-tuning-operator/tuned-xfvkg","openshift-monitoring/node-exporter-qmc6j","openshift-console/downloads-6dd789c8b7-djc8v"] Oct 30 21:43:11.072935 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.072681 5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:11.082425 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.082394 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.097386 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.097362 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.098779 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.098737 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pvwl\"" Oct 30 21:43:11.098980 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.098954 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qtzdm\"" Oct 30 21:43:11.099088 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099073 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:43:11.099157 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099096 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-msl6j\"" Oct 30 21:43:11.099157 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099138 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpjmc\"" Oct 30 21:43:11.099529 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099508 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.099615 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099573 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.099672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099617 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:43:11.099740 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099719 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.099794 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099762 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:43:11.099846 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099829 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.099846 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099837 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:43:11.099959 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099945 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.100008 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.099981 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:43:11.100060 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100020 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.100060 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100057 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.100154 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100071 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.100154 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100094 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:43:11.100154 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100135 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qdnct\"" Oct 30 21:43:11.100154 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100144 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.100154 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100056 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.100428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100161 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xw9dh\"" Oct 30 21:43:11.100428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100210 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fxktb\"" Oct 30 21:43:11.100428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100329 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.100712 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100693 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcz6k\"" Oct 30 21:43:11.100786 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.100742 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:43:11.101500 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.101460 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:43:11.101500 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.101480 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:43:11.101640 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.101507 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:43:11.101640 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.101556 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.102072 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.102051 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:43:11.102185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.102083 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rfsrz\"" Oct 30 21:43:11.102327 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.102212 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:43:11.102327 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.102238 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4c9l\"" Oct 30 21:43:11.102327 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.102263 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vmmq\"" Oct 30 21:43:11.102848 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.102831 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.103055 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.103034 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.103055 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.103044 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.103205 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.103151 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:43:11.103205 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.103199 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:43:11.103284 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.103218 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.114902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.114878 5508 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:43:11.148938 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.148889 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.149130 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149101 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.149283 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149270 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.149425 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149408 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:43:11.149490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149440 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 21:43:11.149785 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149703 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:43:11.149785 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149709 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:43:11.149785 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149764 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.149980 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149796 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:43:11.149980 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149860 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s2czt\"" Oct 30 21:43:11.149980 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.149968 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150008 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150008 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150052 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150095 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150102 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150111 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150148 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:43:11.150185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150184 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 21:43:11.150530 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150227 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-spllf\"" Oct 30 21:43:11.150530 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150233 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 21:43:11.150616 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150536 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:43:11.150616 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.150604 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:43:11.151197 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.151179 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:43:11.151496 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.151466 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:43:11.152170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152152 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6j5v7\"" Oct 30 21:43:11.152352 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152333 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xc8xg\"" Oct 30 21:43:11.152471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152429 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:43:11.152591 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152490 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbz5v\"" Oct 30 21:43:11.152737 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152717 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:43:11.152737 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152737 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hln5f\"" Oct 30 21:43:11.152934 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152754 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tpd8h\"" Oct 30 21:43:11.152934 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152826 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:43:11.152934 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152839 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:43:11.152934 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152844 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dzj62\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152937 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2qslq\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152973 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.152982 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.153000 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-4t47c\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.153052 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9juclv275bcii\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.153085 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-77bv7\"" Oct 30 21:43:11.153125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.153111 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:43:11.158265 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.158244 5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:43:11.158400 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:11.158316 5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-73.us-west-1.compute.internal" Oct 30 21:43:11.159395 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159345 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:43:11.159518 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159393 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:43:11.159518 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159445 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:43:11.159629 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159580 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:43:11.159629 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159613 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pdrmb\"" Oct 30 21:43:11.159727 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159640 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nntrf\"" Oct 30 21:43:11.159727 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159649 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:43:11.159727 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159640 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:43:11.159727 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.159690 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:43:11.197913 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.197891 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:43:11.198003 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.197922 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:43:11.200226 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.200206 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:43:11.201777 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.201758 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:43:11.212384 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212361 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-secret-metrics-server-client-certs\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.212471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212401 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-registry-tls\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.212471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212429 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-etc-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.212471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212456 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-conf-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.212624 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212504 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.212624 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212584 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5574f\" (UniqueName: \"kubernetes.io/projected/e7e07f32-5013-4d0d-ac94-bd8759bfe082-kube-api-access-5574f\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.212704 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212664 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.212760 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212709 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb80ad79-d9ca-41d8-a676-8844585591d4-host-slash\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.212818 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212770 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-registry-tls\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.212818 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212770 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70d8105d-db42-4ae1-9265-028ab1488148-ca-trust-extracted\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.212888 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212770 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bacb62d-563f-428d-9bf1-25b14d6e59c0-metrics-tls\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.212888 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212851 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70d8105d-db42-4ae1-9265-028ab1488148-ca-trust-extracted\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.212991 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212907 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-etc-kubernetes\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.212991 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212951 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.213086 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.212985 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-socket-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.213086 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213019 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-root\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.213086 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213053 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-systemd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.213235 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213096 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-var-lib-kubelet\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.213235 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213140 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-log-socket\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.213235 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213192 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.213396 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213239 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48ea24ed-9083-4214-ac43-06a49f4611dc-crio-socket\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.213396 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213271 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-key\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:43:11.213396 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213374 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ht2x\" (UniqueName: \"kubernetes.io/projected/b59e3ce2-a114-414a-bfdd-23a58df02c91-kube-api-access-6ht2x\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:43:11.213542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213428 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8vd\" (UniqueName: \"kubernetes.io/projected/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kube-api-access-fm8vd\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.213542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213454 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-serving-certs-ca-bundle\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.213542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213478 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d2e278-5632-4d36-87d8-69023ae58aa0-cert\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:43:11.213542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213502 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f749v\" (UniqueName: \"kubernetes.io/projected/6b6edcdf-aa04-4afd-870c-4670f562dc06-kube-api-access-f749v\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:11.213542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213526 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-key\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:43:11.213542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213526 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-trusted-ca\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.213859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213702 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-systemd-units\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.213859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213730 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-textfile\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.213859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213786 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.213859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213817 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-audit-log\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.213859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213828 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-textfile\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.213859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213844 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-multus-certs\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213866 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2d2e278-5632-4d36-87d8-69023ae58aa0-cert\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213885 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-metrics-client-ca\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213937 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthkw\" (UniqueName: \"kubernetes.io/projected/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-kube-api-access-hthkw\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213970 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-kubelet\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.213996 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214033 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ct78\" (UniqueName: \"kubernetes.io/projected/3a0630d9-1d24-4533-ac24-c19bab683ae3-kube-api-access-6ct78\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.214139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214129 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-metrics-client-ca\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214173 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-trusted-ca\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214191 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-host\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214218 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214345 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-slash\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214383 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-registration-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214413 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bacb62d-563f-428d-9bf1-25b14d6e59c0-tmp-dir\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214440 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb80ad79-d9ca-41d8-a676-8844585591d4-iptables-alerter-script\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.214474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214465 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-etc-tuned\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214548 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-system-cni-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214575 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bacb62d-563f-428d-9bf1-25b14d6e59c0-tmp-dir\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214589 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-netns\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214610 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-etc-tuned\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214631 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-var-lib-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214671 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-kubernetes\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214690 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214718 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214725 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb80ad79-d9ca-41d8-a676-8844585591d4-iptables-alerter-script\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214756 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-metrics-client-ca\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.214790 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214790 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cb39913-45bd-4e03-8a53-e5a269238d4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cvlq5\" (UID: \"6cb39913-45bd-4e03-8a53-e5a269238d4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214829 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214902 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b59e3ce2-a114-414a-bfdd-23a58df02c91-signing-cabundle\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214940 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-systemd\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214970 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-run\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.214994 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215015 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-bin\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215039 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215044 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cb39913-45bd-4e03-8a53-e5a269238d4e-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cvlq5\" (UID: \"6cb39913-45bd-4e03-8a53-e5a269238d4e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215104 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215142 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvv6\" (UniqueName: \"kubernetes.io/projected/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-kube-api-access-pwvv6\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215214 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215200 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-bin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.215319 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215276 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-multus\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215329 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-hostroot\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215361 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-federate-client-tls\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215393 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9kb\" (UniqueName: \"kubernetes.io/projected/50ebddb7-2e87-4a26-844e-bcb12013c7c2-kube-api-access-8q9kb\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215431 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-client-ca-bundle\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215430 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215474 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215515 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2cef57-0b45-422b-9390-ee103989900e-hosts-file\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215583 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-metrics-server-audit-profiles\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215645 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-dbus\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215678 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bacb62d-563f-428d-9bf1-25b14d6e59c0-config-volume\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215704 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm2b\" (UniqueName: \"kubernetes.io/projected/b2d2e278-5632-4d36-87d8-69023ae58aa0-kube-api-access-rwm2b\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215772 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-tls\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215815 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysconfig\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.215886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215881 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8bacb62d-563f-428d-9bf1-25b14d6e59c0-config-volume\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.215908 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4nvs\" (UniqueName: \"kubernetes.io/projected/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-kube-api-access-p4nvs\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216003 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-tls\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216051 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovn-node-metrics-cert\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216094 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-system-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216169 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d86f6ae9-bad4-41c8-be3c-61e4225a0501-serviceca\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216246 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnhk\" (UniqueName: \"kubernetes.io/projected/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-kube-api-access-vsnhk\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216358 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-sys-fs\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216359 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovn-node-metrics-cert\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216386 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216424 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-node-log\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216538 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-secret-telemeter-client\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.216589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216581 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-modprobe-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216633 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216712 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-k8s-cni-cncf-io\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216747 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft54x\" (UniqueName: \"kubernetes.io/projected/eb2cef57-0b45-422b-9390-ee103989900e-kube-api-access-ft54x\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216795 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-telemeter-client-tls\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216833 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216844 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216865 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216879 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d86f6ae9-bad4-41c8-be3c-61e4225a0501-serviceca\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216893 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-bound-sa-token\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216935 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2dz\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-kube-api-access-hq2dz\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.216988 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-ovn\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217014 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-env-overrides\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217038 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-cni-binary-copy\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217069 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-device-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217066 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50ebddb7-2e87-4a26-844e-bcb12013c7c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.217138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217128 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-tmp\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217194 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217232 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-conf\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217258 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-sys\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217200 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41738995-836d-4314-baa7-d2e7f5149a56-tmp\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217258 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-cni-binary-copy\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217354 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-installation-pull-secrets\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217388 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-cnibin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217400 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217423 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-env-overrides\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217412 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2cef57-0b45-422b-9390-ee103989900e-tmp-dir\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217462 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb2cef57-0b45-422b-9390-ee103989900e-tmp-dir\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217468 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217562 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-installation-pull-secrets\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.217898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217736 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48ea24ed-9083-4214-ac43-06a49f4611dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217526 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxq22\" (UniqueName: \"kubernetes.io/projected/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-api-access-dxq22\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.217990 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zqr\" (UniqueName: \"kubernetes.io/projected/eb80ad79-d9ca-41d8-a676-8844585591d4-kube-api-access-s4zqr\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218020 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218046 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d86f6ae9-bad4-41c8-be3c-61e4225a0501-host\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218066 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-accelerators-collector-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218116 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218145 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-socket-dir-parent\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218171 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcp62\" (UniqueName: \"kubernetes.io/projected/9bc0c153-0125-47d4-bf2d-27668920eb0b-kube-api-access-tcp62\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.218492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218196 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.218952 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218603 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-accelerators-collector-config\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.218952 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218634 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cnibin\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.218952 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218930 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.219109 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.218992 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-image-registry-private-configuration\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.219109 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219101 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5f8\" (UniqueName: \"kubernetes.io/projected/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-kube-api-access-6m5f8\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:11.219211 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219133 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.219347 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219221 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-agent-certs\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:43:11.219449 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219418 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-wtmp\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.219630 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219455 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:11.219630 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219487 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.219630 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219564 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-netd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.219630 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219598 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-script-lib\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.219630 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219628 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-kubelet\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.219881 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219659 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-konnectivity-ca\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:43:11.219881 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219683 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-sys\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.220079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.219981 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.220079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220018 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-lib-modules\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.220079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220026 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-agent-certs\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:43:11.220079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220052 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220091 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220137 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-registry-certificates\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220180 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-config\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220209 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-daemon-config\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220461 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04ecd634-c544-41e3-82bd-b5e36a5d57cd-original-pull-secret\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220492 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcgb\" (UniqueName: \"kubernetes.io/projected/d86f6ae9-bad4-41c8-be3c-61e4225a0501-kube-api-access-qvcgb\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220532 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-tmp\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220568 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-kubelet-config\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220602 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ebddb7-2e87-4a26-844e-bcb12013c7c2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.220759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220758 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-script-lib\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.221259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220799 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-secret-metrics-server-tls\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.221259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220839 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-netns\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.221259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220876 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r959h\" (UniqueName: \"kubernetes.io/projected/8bacb62d-563f-428d-9bf1-25b14d6e59c0-kube-api-access-r959h\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.221259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220909 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7xz\" (UniqueName: \"kubernetes.io/projected/41738995-836d-4314-baa7-d2e7f5149a56-kube-api-access-rp7xz\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.221259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220935 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc-konnectivity-ca\") pod \"konnectivity-agent-cggst\" (UID: \"1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc\") " pod="kube-system/konnectivity-agent-cggst" Oct 30 21:43:11.221259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.220943 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ea24ed-9083-4214-ac43-06a49f4611dc-data-volume\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.222037 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.221900 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.222037 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.221981 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48ea24ed-9083-4214-ac43-06a49f4611dc-data-volume\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.222395 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.222162 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-daemon-config\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.222395 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.222381 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.222729 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.222576 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70d8105d-db42-4ae1-9265-028ab1488148-image-registry-private-configuration\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.222856 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.222801 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6edcdf-aa04-4afd-870c-4670f562dc06-metrics-certs\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:11.223052 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.223013 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70d8105d-db42-4ae1-9265-028ab1488148-registry-certificates\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.223262 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.223243 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-ovnkube-config\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.223873 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.223850 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-tmp\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:11.224398 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.224194 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50ebddb7-2e87-4a26-844e-bcb12013c7c2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.269648 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.269608 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ct78\" (UniqueName: \"kubernetes.io/projected/3a0630d9-1d24-4533-ac24-c19bab683ae3-kube-api-access-6ct78\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.269818 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.269716 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f749v\" (UniqueName: \"kubernetes.io/projected/6b6edcdf-aa04-4afd-870c-4670f562dc06-kube-api-access-f749v\") pod \"network-metrics-daemon-bpfn7\" (UID: \"6b6edcdf-aa04-4afd-870c-4670f562dc06\") " pod="openshift-multus/network-metrics-daemon-bpfn7" Oct 30 21:43:11.269945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.269927 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8vd\" (UniqueName: \"kubernetes.io/projected/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kube-api-access-fm8vd\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.270003 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.269927 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ht2x\" (UniqueName: \"kubernetes.io/projected/b59e3ce2-a114-414a-bfdd-23a58df02c91-kube-api-access-6ht2x\") pod \"service-ca-5fffc97bdf-7qf6l\" (UID: \"b59e3ce2-a114-414a-bfdd-23a58df02c91\") " pod="openshift-service-ca/service-ca-5fffc97bdf-7qf6l" Oct 30 21:43:11.270575 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.270552 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvv6\" (UniqueName: \"kubernetes.io/projected/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-kube-api-access-pwvv6\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.273288 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.273262 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm2b\" (UniqueName: \"kubernetes.io/projected/b2d2e278-5632-4d36-87d8-69023ae58aa0-kube-api-access-rwm2b\") pod \"ingress-canary-jfjmh\" (UID: \"b2d2e278-5632-4d36-87d8-69023ae58aa0\") " pod="openshift-ingress-canary/ingress-canary-jfjmh" Oct 30 21:43:11.273417 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.273291 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnhk\" (UniqueName: \"kubernetes.io/projected/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-kube-api-access-vsnhk\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.273497 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.273479 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9kb\" (UniqueName: \"kubernetes.io/projected/50ebddb7-2e87-4a26-844e-bcb12013c7c2-kube-api-access-8q9kb\") pod \"prometheus-operator-9cd89b74c-gtjhm\" (UID: \"50ebddb7-2e87-4a26-844e-bcb12013c7c2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-gtjhm" Oct 30 21:43:11.273866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.273842 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2dz\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-kube-api-access-hq2dz\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.274240 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.274221 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4nvs\" (UniqueName: \"kubernetes.io/projected/ccb804c3-d2fb-48bb-be14-3b1c6743f49f-kube-api-access-p4nvs\") pod \"openshift-state-metrics-7bd7489957-fbgs6\" (UID: \"ccb804c3-d2fb-48bb-be14-3b1c6743f49f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fbgs6" Oct 30 21:43:11.275230 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.275201 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxq22\" (UniqueName: \"kubernetes.io/projected/48ea24ed-9083-4214-ac43-06a49f4611dc-kube-api-access-dxq22\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.275410 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.275391 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70d8105d-db42-4ae1-9265-028ab1488148-bound-sa-token\") pod \"image-registry-5f7d486fcd-vl4dg\" (UID: \"70d8105d-db42-4ae1-9265-028ab1488148\") " pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:11.276562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.276540 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft54x\" (UniqueName: \"kubernetes.io/projected/eb2cef57-0b45-422b-9390-ee103989900e-kube-api-access-ft54x\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.276878 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.276857 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6z2d\" (UniqueName: \"kubernetes.io/projected/f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b-kube-api-access-w6z2d\") pod \"network-check-target-ng9zm\" (UID: \"f3b71dff-9f5e-4722-8ee4-eb5dc2a2ca2b\") " pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:43:11.277419 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.277370 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zqr\" (UniqueName: \"kubernetes.io/projected/eb80ad79-d9ca-41d8-a676-8844585591d4-kube-api-access-s4zqr\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.277419 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.277409 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcp62\" (UniqueName: \"kubernetes.io/projected/9bc0c153-0125-47d4-bf2d-27668920eb0b-kube-api-access-tcp62\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.277684 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.277665 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5f8\" (UniqueName: \"kubernetes.io/projected/03cbfbcf-0c69-4649-a966-c29bb2ef0d05-kube-api-access-6m5f8\") pod \"downloads-6dd789c8b7-djc8v\" (UID: \"03cbfbcf-0c69-4649-a966-c29bb2ef0d05\") " pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:11.279490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.279469 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcgb\" (UniqueName: \"kubernetes.io/projected/d86f6ae9-bad4-41c8-be3c-61e4225a0501-kube-api-access-qvcgb\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.280348 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.280328 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7xz\" (UniqueName: \"kubernetes.io/projected/41738995-836d-4314-baa7-d2e7f5149a56-kube-api-access-rp7xz\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.280682 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.280663 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r959h\" (UniqueName: \"kubernetes.io/projected/8bacb62d-563f-428d-9bf1-25b14d6e59c0-kube-api-access-r959h\") pod \"dns-default-nwqgt\" (UID: \"8bacb62d-563f-428d-9bf1-25b14d6e59c0\") " pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:11.322355 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322326 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-secret-metrics-server-tls\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.322477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322359 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-netns\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322388 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-secret-metrics-server-client-certs\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.322477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322414 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-etc-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.322477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322435 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-conf-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322459 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5574f\" (UniqueName: \"kubernetes.io/projected/e7e07f32-5013-4d0d-ac94-bd8759bfe082-kube-api-access-5574f\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322482 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322491 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-conf-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322458 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-netns\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322505 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb80ad79-d9ca-41d8-a676-8844585591d4-host-slash\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322488 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-etc-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322528 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-etc-kubernetes\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322547 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb80ad79-d9ca-41d8-a676-8844585591d4-host-slash\") pod \"iptables-alerter-5kndd\" (UID: \"eb80ad79-d9ca-41d8-a676-8844585591d4\") " pod="openshift-network-operator/iptables-alerter-5kndd" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322552 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322582 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-socket-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322599 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322607 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-root\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322619 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-etc-kubernetes\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322629 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-systemd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322650 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-socket-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322661 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-systemd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322667 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-var-lib-kubelet\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322690 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-root\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322694 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-log-socket\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.322713 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322707 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-var-lib-kubelet\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322726 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-log-socket\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322727 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322756 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322764 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48ea24ed-9083-4214-ac43-06a49f4611dc-crio-socket\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322798 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-serving-certs-ca-bundle\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322806 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48ea24ed-9083-4214-ac43-06a49f4611dc-crio-socket\") pod \"insights-runtime-extractor-lbksg\" (UID: \"48ea24ed-9083-4214-ac43-06a49f4611dc\") " pod="openshift-insights/insights-runtime-extractor-lbksg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322818 5508 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322829 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-systemd-units\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322854 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-audit-log\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322880 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-multus-certs\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322904 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthkw\" (UniqueName: \"kubernetes.io/projected/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-kube-api-access-hthkw\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322917 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-systemd-units\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322928 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-kubelet\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322952 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.322976 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-host\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323005 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-slash\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323028 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-registration-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323056 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-system-cni-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323079 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-netns\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323102 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-var-lib-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323125 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-kubernetes\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323150 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323175 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-metrics-client-ca\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323200 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323223 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-systemd\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323238 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-audit-log\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323246 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-run\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323258 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-multus-certs\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323287 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-bin\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323331 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-netns\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323356 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-host\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323359 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323388 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-slash\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323410 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323415 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-bin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323442 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-multus\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323445 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-registration-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323455 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-var-lib-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323483 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-system-cni-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323511 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-hostroot\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323520 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-kubelet\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323560 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-openvswitch\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323579 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-multus\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323584 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-serving-certs-ca-bundle\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323602 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-federate-client-tls\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323609 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-hostroot\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323661 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-cni-bin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323662 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-kubernetes\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.323866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323857 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323921 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-run\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.323958 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-bin\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324073 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-client-ca-bundle\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324130 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-systemd\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324185 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324294 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324377 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324245 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2cef57-0b45-422b-9390-ee103989900e-hosts-file\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324531 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb2cef57-0b45-422b-9390-ee103989900e-hosts-file\") pod \"node-resolver-tsn5z\" (UID: \"eb2cef57-0b45-422b-9390-ee103989900e\") " pod="openshift-dns/node-resolver-tsn5z" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324475 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-metrics-server-audit-profiles\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.324581 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-dbus\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325529 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-metrics-server-audit-profiles\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325591 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysconfig\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325674 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-dbus\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325711 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-system-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325790 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysconfig\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325840 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-system-cni-dir\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325743 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-sys-fs\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325897 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-node-log\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325964 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-sys-fs\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.325995 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-secret-telemeter-client\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326000 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-node-log\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326019 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-modprobe-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326063 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-k8s-cni-cncf-io\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.326188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326193 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-modprobe-d\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326236 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-run-k8s-cni-cncf-io\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326088 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-telemeter-client-tls\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326281 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326347 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-metrics-client-ca\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326620 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-ovn\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326664 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-device-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326691 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-conf\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326716 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-sys\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326740 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-cnibin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326769 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326792 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d86f6ae9-bad4-41c8-be3c-61e4225a0501-host\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326817 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-socket-dir-parent\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326841 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cnibin\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326869 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-wtmp\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326896 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-netd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326919 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-kubelet\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326941 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-sys\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326964 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-lib-modules\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.326989 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327020 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-kubelet-config\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327123 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04ecd634-c544-41e3-82bd-b5e36a5d57cd-kubelet-config\") pod \"global-pull-secret-syncer-56vhn\" (UID: \"04ecd634-c544-41e3-82bd-b5e36a5d57cd\") " pod="kube-system/global-pull-secret-syncer-56vhn" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327183 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327224 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-run-ovn\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327272 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe3573ec-d74e-44d9-aec5-ca58f51e04fd-device-dir\") pod \"aws-ebs-csi-driver-node-8hfct\" (UID: \"fe3573ec-d74e-44d9-aec5-ca58f51e04fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hfct" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327347 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-secret-metrics-server-client-certs\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327421 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5e31250d-f97f-4b8b-9836-2f77a5deb9ce-cnibin\") pod \"multus-additional-cni-plugins-8hwrm\" (UID: \"5e31250d-f97f-4b8b-9836-2f77a5deb9ce\") " pod="openshift-multus/multus-additional-cni-plugins-8hwrm" Oct 30 21:43:11.327428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327437 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-host-var-lib-kubelet\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327476 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-sys\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327496 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-node-exporter-wtmp\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327527 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-cnibin\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327536 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e-host-cni-netd\") pod \"ovnkube-node-7mj55\" (UID: \"6ec68d72-5b2e-4f80-8ae2-4fddcdd3d55e\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327582 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d86f6ae9-bad4-41c8-be3c-61e4225a0501-host\") pod \"node-ca-2r78l\" (UID: \"d86f6ae9-bad4-41c8-be3c-61e4225a0501\") " pod="openshift-image-registry/node-ca-2r78l" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327631 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc0c153-0125-47d4-bf2d-27668920eb0b-multus-socket-dir-parent\") pod \"multus-qg64f\" (UID: \"9bc0c153-0125-47d4-bf2d-27668920eb0b\") " pod="openshift-multus/multus-qg64f" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327731 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-lib-modules\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.327773 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0630d9-1d24-4533-ac24-c19bab683ae3-sys\") pod \"node-exporter-qmc6j\" (UID: \"3a0630d9-1d24-4533-ac24-c19bab683ae3\") " pod="openshift-monitoring/node-exporter-qmc6j" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.328136 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.328673 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7e07f32-5013-4d0d-ac94-bd8759bfe082-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.328890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.328741 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/41738995-836d-4314-baa7-d2e7f5149a56-etc-sysctl-conf\") pod \"tuned-xfvkg\" (UID: \"41738995-836d-4314-baa7-d2e7f5149a56\") " pod="openshift-cluster-node-tuning-operator/tuned-xfvkg" Oct 30 21:43:11.329969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.329946 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-client-ca-bundle\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.330330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.330289 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-secret-telemeter-client\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.330503 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.330476 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-secret-metrics-server-tls\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.330927 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.330898 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.331102 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.331069 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-federate-client-tls\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.332834 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.332812 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e7e07f32-5013-4d0d-ac94-bd8759bfe082-telemeter-client-tls\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.385601 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.385568 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthkw\" (UniqueName: \"kubernetes.io/projected/26c4ad3d-7555-4bbf-834e-299ef37ffbcf-kube-api-access-hthkw\") pod \"metrics-server-6499d46c6f-54lvw\" (UID: \"26c4ad3d-7555-4bbf-834e-299ef37ffbcf\") " pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.385892 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.385868 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5574f\" (UniqueName: \"kubernetes.io/projected/e7e07f32-5013-4d0d-ac94-bd8759bfe082-kube-api-access-5574f\") pod \"telemeter-client-5659774bc7-pwf5k\" (UID: \"e7e07f32-5013-4d0d-ac94-bd8759bfe082\") " pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.406619 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.406566 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:11.452698 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.452666 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" Oct 30 21:43:11.865482 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.865449 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6499d46c6f-54lvw"] Oct 30 21:43:11.870352 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:11.870297 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c4ad3d_7555_4bbf_834e_299ef37ffbcf.slice/crio-8ea201153c699c864b65682f16ad2c2f10d143607a0794bf610ca694338117a4 WatchSource:0}: Error finding container 8ea201153c699c864b65682f16ad2c2f10d143607a0794bf610ca694338117a4: Status 404 returned error can't find the container with id 8ea201153c699c864b65682f16ad2c2f10d143607a0794bf610ca694338117a4 Oct 30 21:43:11.873021 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.873001 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:43:11.922443 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:11.922393 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5659774bc7-pwf5k"] Oct 30 21:43:11.926794 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:11.926761 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e07f32_5013_4d0d_ac94_bd8759bfe082.slice/crio-f5b4f7cb7fa0670126bd987c94d78150384638a896793c456a80b3be2dc71ba8 WatchSource:0}: Error finding container f5b4f7cb7fa0670126bd987c94d78150384638a896793c456a80b3be2dc71ba8: Status 404 returned error can't find the container with id f5b4f7cb7fa0670126bd987c94d78150384638a896793c456a80b3be2dc71ba8 Oct 30 21:43:12.076472 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:12.076434 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" event={"ID":"e7e07f32-5013-4d0d-ac94-bd8759bfe082","Type":"ContainerStarted","Data":"f5b4f7cb7fa0670126bd987c94d78150384638a896793c456a80b3be2dc71ba8"} Oct 30 21:43:12.077890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:12.077854 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" event={"ID":"26c4ad3d-7555-4bbf-834e-299ef37ffbcf","Type":"ContainerStarted","Data":"8ea201153c699c864b65682f16ad2c2f10d143607a0794bf610ca694338117a4"} Oct 30 21:43:13.455645 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:13.454985 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nwqgt" Oct 30 21:43:13.651250 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:13.650640 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" Oct 30 21:43:14.064975 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.064947 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:14.182189 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.182153 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:43:14.201600 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.201559 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.209441 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.208415 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252485 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252245 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252485 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252289 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gswcq\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252485 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252336 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252485 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252369 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252515 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252544 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252572 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252602 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252647 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252676 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252705 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252738 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252778 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.252801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252800 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.253191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252828 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.253191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252858 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.253191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252897 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.253191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252921 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.253191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252949 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.253191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.252979 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.268362 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.262785 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:43:14.314891 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.314815 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:43:14.315121 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.314904 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:43:14.315121 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315019 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rn5mz\"" Oct 30 21:43:14.315267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315121 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-n62a1hpdvb0v\"" Oct 30 21:43:14.315267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315159 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:43:14.315267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315259 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:43:14.315470 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315394 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:43:14.315470 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315463 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:43:14.315579 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315525 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:43:14.315635 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315609 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:43:14.315719 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315706 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:43:14.315801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315777 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:43:14.316010 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315908 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:43:14.316010 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.315965 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353669 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353782 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353824 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353874 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353911 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353972 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.353997 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354046 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354077 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354106 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354121 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354140 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354160 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354192 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354215 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswcq\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354243 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354269 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354297 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354340 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.354474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354381 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.355447 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.354808 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.355895 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.355645 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.356739 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.356390 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.356739 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.356631 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.357163 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.357139 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.357852 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.357825 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.358478 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.358345 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.359854 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.359493 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.362237 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.362216 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.362836 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.362762 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.363216 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.363196 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.367254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.367207 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.367876 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.367824 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.367876 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.367859 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.368523 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.368455 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.368523 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.368457 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.371135 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.371092 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.398509 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.397422 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ng9zm" Oct 30 21:43:14.398509 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.398442 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.413974 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.413941 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:43:14.414330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.414285 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:43:14.416129 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.415712 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.417581 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.417535 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswcq\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.420644 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.420617 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:14.582626 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.582542 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:14.642796 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:14.642594 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:15.101588 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:15.101551 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" event={"ID":"26c4ad3d-7555-4bbf-834e-299ef37ffbcf","Type":"ContainerStarted","Data":"1c297a874c33f25d38fe23c69d924cd05e89d8c750b74f09664f044a730f60a1"} Oct 30 21:43:15.162474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:15.162445 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:43:15.218832 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:15.218728 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" podStartSLOduration=3.132461951 podStartE2EDuration="6.218710522s" totalImagesPullingTime="3.086248571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:15.217485618 +0000 UTC m=+5.407023293" watchObservedRunningTime="2026-10-30 21:43:15.218710522 +0000 UTC m=+5.408248199" Oct 30 21:43:15.265562 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:43:15.265531 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d7f1a2_5634_44da_9108_4f1e08a232a0.slice/crio-6011d4ee2fac78bb7df8ab0d2437d96ca68ba34e42cc3f166c7d0fb16c1c9188 WatchSource:0}: Error finding container 6011d4ee2fac78bb7df8ab0d2437d96ca68ba34e42cc3f166c7d0fb16c1c9188: Status 404 returned error can't find the container with id 6011d4ee2fac78bb7df8ab0d2437d96ca68ba34e42cc3f166c7d0fb16c1c9188 Oct 30 21:43:15.296707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:15.296681 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:16.106820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:16.106768 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" event={"ID":"e7e07f32-5013-4d0d-ac94-bd8759bfe082","Type":"ContainerStarted","Data":"d50e79e7eb29883cfe0051e63ace49cb744b7fdd21c3ce0f6bf5f6cc232562c6"} Oct 30 21:43:16.107941 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:16.107918 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"6011d4ee2fac78bb7df8ab0d2437d96ca68ba34e42cc3f166c7d0fb16c1c9188"} Oct 30 21:43:17.240801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:17.240775 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:17.450741 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:17.450708 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:18.117476 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:18.117443 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" event={"ID":"e7e07f32-5013-4d0d-ac94-bd8759bfe082","Type":"ContainerStarted","Data":"a4d06e5b1992b913d21d29326b01e00f1584649b3cdc79eb012c535f5e4ce8fb"} Oct 30 21:43:18.117715 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:18.117508 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" event={"ID":"e7e07f32-5013-4d0d-ac94-bd8759bfe082","Type":"ContainerStarted","Data":"3c4ff2a8e38a258f9bec317adac2224149c9a6223d7fc26bd9c8df0afc76a5b6"} Oct 30 21:43:18.118995 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:18.118965 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" exitCode=0 Oct 30 21:43:18.119118 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:18.119005 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} Oct 30 21:43:18.240124 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:18.240078 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5659774bc7-pwf5k" podStartSLOduration=3.047442652 podStartE2EDuration="8.240065136s" totalImagesPullingTime="5.192622484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:18.238960419 +0000 UTC m=+8.428498094" watchObservedRunningTime="2026-10-30 21:43:18.240065136 +0000 UTC m=+8.429602805" Oct 30 21:43:19.073209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.073172 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:19.274556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.274518 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:19.302830 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.302792 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:19.303079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.303056 5508 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:43:19.436436 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.436355 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:19.439171 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.439145 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-djc8v" Oct 30 21:43:19.874694 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:19.874635 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:20.058174 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:20.058143 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:20.117208 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:20.117181 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f7d486fcd-vl4dg" Oct 30 21:43:20.419869 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:20.419835 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:20.614263 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:20.614211 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:20.873808 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:20.873767 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:21.968701 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:21.968663 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.130029 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:22.129982 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:22.247337 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:22.247297 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cggst" Oct 30 21:43:23.137462 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:23.137435 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} Oct 30 21:43:23.137817 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:23.137498 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} Oct 30 21:43:23.857509 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:23.857433 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:24.018541 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:24.018505 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:26.151706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:26.151670 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} Oct 30 21:43:26.152194 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:26.151734 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} Oct 30 21:43:26.152194 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:26.151758 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} Oct 30 21:43:26.152194 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:26.151783 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerStarted","Data":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} Oct 30 21:43:26.434390 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:26.434263 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.74662914 podStartE2EDuration="12.434250858s" totalImagesPullingTime="8.654043955s" totalInitContainerRuntime="33.577763ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:26.384875293 +0000 UTC m=+16.574412970" watchObservedRunningTime="2026-10-30 21:43:26.434250858 +0000 UTC m=+16.623788528" Oct 30 21:43:28.012949 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:28.012917 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:29.656370 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:29.656339 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mj55" Oct 30 21:43:30.951676 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:30.951648 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:31.087634 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:31.087603 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:31.250456 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:31.250387 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:33.229557 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:33.229523 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:34.068076 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:34.068044 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:34.251029 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:34.250994 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:38.230938 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:38.230913 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:38.249113 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:38.249090 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:39.216751 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:39.216724 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:43:40.371563 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:43:40.371524 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:43:41.906060 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:41.906027 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:49.104219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:49.104183 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:49.291120 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:49.291086 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:50.957979 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:50.957948 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:50.961885 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:50.961866 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6499d46c6f-54lvw" Oct 30 21:43:52.805501 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:52.805457 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:56.863251 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:56.863212 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.069376 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:57.069346 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.271121 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:57.271035 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:57.465440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:43:57.465405 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:05.591988 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:05.591950 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:05.755648 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:05.755613 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:06.283125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.283091 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:44:06.283589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.283541 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="prometheus" containerID="cri-o://f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" gracePeriod=600 Oct 30 21:44:06.284254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.283742 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" gracePeriod=600 Oct 30 21:44:06.284254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.283797 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="thanos-sidecar" containerID="cri-o://1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" gracePeriod=600 Oct 30 21:44:06.284254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.283846 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="config-reloader" containerID="cri-o://964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" gracePeriod=600 Oct 30 21:44:06.284254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.283927 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy" containerID="cri-o://43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" gracePeriod=600 Oct 30 21:44:06.284254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.284008 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-web" containerID="cri-o://adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" gracePeriod=600 Oct 30 21:44:06.531772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.531748 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:06.606341 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606292 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606356 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606385 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606427 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606463 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606493 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606531 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606557 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606591 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606622 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606651 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606685 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606712 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606746 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.606820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606774 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606831 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606858 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606875 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606893 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606922 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606927 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.606965 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db\") pod \"10d7f1a2-5634-44da-9108-4f1e08a232a0\" (UID: \"10d7f1a2-5634-44da-9108-4f1e08a232a0\") " Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.607205 5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.607524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.607222 5508 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.607972 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.607847 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.608025 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.607979 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.608639 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.608602 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.608966 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.608927 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.609875 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.609845 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:06.609969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.609944 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.610034 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.610013 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "kube-api-access-gswcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:06.610646 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.610602 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:06.610737 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.610668 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.610836 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.610812 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.611803 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.611762 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.611803 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.611776 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.611931 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.611912 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.612091 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.612069 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:06.612408 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.612386 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:06.612554 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.612535 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.612716 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.612701 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.621872 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.621852 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config" pod "10d7f1a2-5634-44da-9108-4f1e08a232a0" (UID: "10d7f1a2-5634-44da-9108-4f1e08a232a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:06.708582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708549 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gswcq\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-kube-api-access-gswcq\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708578 5508 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-web-config\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708589 5508 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-kube-rbac-proxy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708599 5508 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708610 5508 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10d7f1a2-5634-44da-9108-4f1e08a232a0-tls-assets\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708618 5508 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-config\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708626 5508 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708634 5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708642 5508 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-config-out\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708651 5508 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-grpc-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708658 5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708669 5508 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708678 5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708686 5508 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-metrics-client-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708694 5508 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10d7f1a2-5634-44da-9108-4f1e08a232a0-configmap-metrics-client-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708704 5508 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708713 5508 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10d7f1a2-5634-44da-9108-4f1e08a232a0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:06.708792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:06.708721 5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10d7f1a2-5634-44da-9108-4f1e08a232a0-prometheus-k8s-db\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:44:07.303000 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.302967 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" exitCode=0 Oct 30 21:44:07.303000 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.302997 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" exitCode=0 Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303009 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" exitCode=0 Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303018 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" exitCode=0 Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303023 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" exitCode=0 Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303028 5508 generic.go:356] "Generic (PLEG): container finished" podID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" exitCode=0 Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303066 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303099 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303128 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303160 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} Oct 30 21:44:07.303212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303192 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} Oct 30 21:44:07.303605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303231 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} Oct 30 21:44:07.303605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303265 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} Oct 30 21:44:07.303605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303295 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10d7f1a2-5634-44da-9108-4f1e08a232a0","Type":"ContainerDied","Data":"6011d4ee2fac78bb7df8ab0d2437d96ca68ba34e42cc3f166c7d0fb16c1c9188"} Oct 30 21:44:07.303605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.303356 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.315815 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.315795 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.322805 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.322781 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.329200 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.329181 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.335577 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.335556 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.341910 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.341891 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.348574 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.348559 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.354987 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.354972 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.355266 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.355244 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.355336 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.355276 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} err="failed to get container status \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" Oct 30 21:44:07.355386 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.355335 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.355550 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.355522 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.355550 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.355557 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} err="failed to get container status \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" Oct 30 21:44:07.355805 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.355579 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.355857 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.355832 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.355902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.355869 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} err="failed to get container status \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" Oct 30 21:44:07.355902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.355889 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.356170 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.356143 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.356257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.356174 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} err="failed to get container status \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" Oct 30 21:44:07.356257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.356188 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.356569 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.356427 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.356569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.356454 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} err="failed to get container status \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" Oct 30 21:44:07.356569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.356472 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.356737 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.356695 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.356737 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.356721 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} err="failed to get container status \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" Oct 30 21:44:07.356846 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.356736 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.357031 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:07.357013 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.357090 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357034 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} err="failed to get container status \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" Oct 30 21:44:07.357090 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357049 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.357263 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357242 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} err="failed to get container status \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" Oct 30 21:44:07.357347 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357264 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.357513 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357494 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} err="failed to get container status \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" Oct 30 21:44:07.357513 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357511 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.357733 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357713 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} err="failed to get container status \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" Oct 30 21:44:07.357777 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357734 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.357931 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357915 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} err="failed to get container status \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" Oct 30 21:44:07.357975 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.357931 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.358135 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358120 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} err="failed to get container status \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" Oct 30 21:44:07.358172 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358134 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.358355 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358338 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} err="failed to get container status \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" Oct 30 21:44:07.358403 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358355 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.358565 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358550 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} err="failed to get container status \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" Oct 30 21:44:07.358610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358567 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.358769 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358752 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} err="failed to get container status \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" Oct 30 21:44:07.358806 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358769 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.358962 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358947 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} err="failed to get container status \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" Oct 30 21:44:07.359002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.358962 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.359145 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359129 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} err="failed to get container status \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" Oct 30 21:44:07.359184 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359146 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.359371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359352 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} err="failed to get container status \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" Oct 30 21:44:07.359448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359374 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.359598 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359582 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} err="failed to get container status \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" Oct 30 21:44:07.359645 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359598 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.359808 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359790 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} err="failed to get container status \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" Oct 30 21:44:07.359876 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359809 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.360006 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.359991 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} err="failed to get container status \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" Oct 30 21:44:07.360050 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360007 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.360209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360191 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} err="failed to get container status \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" Oct 30 21:44:07.360279 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360210 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.360414 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360397 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} err="failed to get container status \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" Oct 30 21:44:07.360465 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360415 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.360602 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360587 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} err="failed to get container status \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" Oct 30 21:44:07.360602 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360601 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.360813 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360795 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} err="failed to get container status \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" Oct 30 21:44:07.360856 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360813 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.361007 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.360992 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} err="failed to get container status \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" Oct 30 21:44:07.361048 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361006 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.361189 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361172 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} err="failed to get container status \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" Oct 30 21:44:07.361225 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361189 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.361394 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361380 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} err="failed to get container status \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" Oct 30 21:44:07.361445 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361394 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.361575 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361559 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} err="failed to get container status \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" Oct 30 21:44:07.361617 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361575 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.361752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361736 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} err="failed to get container status \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" Oct 30 21:44:07.361792 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361753 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.361997 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361976 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} err="failed to get container status \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" Oct 30 21:44:07.362061 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.361999 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.362207 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362183 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} err="failed to get container status \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" Oct 30 21:44:07.362266 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362207 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.362413 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362397 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} err="failed to get container status \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" Oct 30 21:44:07.362413 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362412 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.362623 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362608 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} err="failed to get container status \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" Oct 30 21:44:07.362678 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362623 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.362829 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362807 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} err="failed to get container status \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" Oct 30 21:44:07.362829 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.362827 5508 scope.go:122] "RemoveContainer" containerID="963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997" Oct 30 21:44:07.363074 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363059 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997"} err="failed to get container status \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": rpc error: code = NotFound desc = could not find container \"963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997\": container with ID starting with 963ca89a4f875b3ee92ef778a18ef515d1dbbb120e8ed053f95ce3bf5b725997 not found: ID does not exist" Oct 30 21:44:07.363074 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363073 5508 scope.go:122] "RemoveContainer" containerID="43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7" Oct 30 21:44:07.363296 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363281 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7"} err="failed to get container status \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": rpc error: code = NotFound desc = could not find container \"43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7\": container with ID starting with 43b35a023c5dba3a41781c26631dad8f6fd969d259651c6b6c2ae69b93e3cbf7 not found: ID does not exist" Oct 30 21:44:07.363296 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363296 5508 scope.go:122] "RemoveContainer" containerID="adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6" Oct 30 21:44:07.363556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363542 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6"} err="failed to get container status \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": rpc error: code = NotFound desc = could not find container \"adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6\": container with ID starting with adb66ab96578786cc4e2dcc6018aa8da620dfbc7792121463f61ed896eb85da6 not found: ID does not exist" Oct 30 21:44:07.363556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363556 5508 scope.go:122] "RemoveContainer" containerID="1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1" Oct 30 21:44:07.363824 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363804 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1"} err="failed to get container status \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": rpc error: code = NotFound desc = could not find container \"1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1\": container with ID starting with 1736ba84178d3c2675e3c11902330c0ed640bd401895c593730bf43e6ddaecb1 not found: ID does not exist" Oct 30 21:44:07.363824 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.363824 5508 scope.go:122] "RemoveContainer" containerID="964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1" Oct 30 21:44:07.364066 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.364049 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1"} err="failed to get container status \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": rpc error: code = NotFound desc = could not find container \"964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1\": container with ID starting with 964c2e22867f3ac86885ce523e096af00e410ed184c8acb8d3d44c63d870b1d1 not found: ID does not exist" Oct 30 21:44:07.364114 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.364066 5508 scope.go:122] "RemoveContainer" containerID="f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7" Oct 30 21:44:07.364288 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.364274 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7"} err="failed to get container status \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": rpc error: code = NotFound desc = could not find container \"f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7\": container with ID starting with f3194eace0452e37455ebb3287ab550ef97ca345085a32550f0a86cac85931a7 not found: ID does not exist" Oct 30 21:44:07.364352 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.364289 5508 scope.go:122] "RemoveContainer" containerID="3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335" Oct 30 21:44:07.364531 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.364511 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335"} err="failed to get container status \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": rpc error: code = NotFound desc = could not find container \"3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335\": container with ID starting with 3d6730090c79361cbe7f23d30204d997b4308ae6c3128ba0ef9039e018deb335 not found: ID does not exist" Oct 30 21:44:07.469948 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.469917 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:44:07.519228 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.519193 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:44:07.526909 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.526883 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:44:07.527164 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527152 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527166 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527174 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-thanos" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527179 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-thanos" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527189 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="prometheus" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527193 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="prometheus" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527203 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="init-config-reloader" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527207 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="init-config-reloader" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527213 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="config-reloader" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527218 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="config-reloader" Oct 30 21:44:07.527219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527224 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="thanos-sidecar" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527229 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="thanos-sidecar" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527241 5508 cpu_manager.go:414] "containerMap: removing container" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-web" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527245 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-web" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527287 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527294 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-thanos" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527300 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="prometheus" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527321 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="config-reloader" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527326 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="thanos-sidecar" Oct 30 21:44:07.527510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.527333 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" containerName="kube-rbac-proxy-web" Oct 30 21:44:07.531114 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.531095 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.583803 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583771 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:44:07.583803 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583771 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583825 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583843 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583874 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583880 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583925 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.583995 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:44:07.584017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.584006 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rn5mz\"" Oct 30 21:44:07.584488 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.584123 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:44:07.584488 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.584130 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:44:07.584488 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.584190 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:44:07.584488 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.584253 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:44:07.615955 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.615931 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.615956 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.615981 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.615999 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/907c5318-2bc1-44cb-89ef-757db10b6009-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616048 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616074 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-web-config\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616123 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616163 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-config\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616241 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616264 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616286 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616313 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616330 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616403 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/907c5318-2bc1-44cb-89ef-757db10b6009-config-out\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616431 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616451 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616468 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616534 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616553 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrhw\" (UniqueName: \"kubernetes.io/projected/907c5318-2bc1-44cb-89ef-757db10b6009-kube-api-access-7mrhw\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.616610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.616572 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.633145 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.633117 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-n62a1hpdvb0v\"" Oct 30 21:44:07.633259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.633193 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:44:07.634059 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.634043 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:44:07.643967 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.643948 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:44:07.717863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.717835 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.717969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.717866 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/907c5318-2bc1-44cb-89ef-757db10b6009-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.717969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.717885 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.717969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.717907 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-web-config\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.717969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.717929 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718216 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718068 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-config\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718216 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718112 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718216 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718140 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718216 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718172 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718555 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718509 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718976 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718891 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718976 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718928 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718976 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718946 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.718976 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.718963 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.719259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719231 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.719424 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719325 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/907c5318-2bc1-44cb-89ef-757db10b6009-config-out\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719601 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719644 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719674 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719715 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719744 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrhw\" (UniqueName: \"kubernetes.io/projected/907c5318-2bc1-44cb-89ef-757db10b6009-kube-api-access-7mrhw\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719777 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719820 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719844 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.719901 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.720480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.720021 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.722448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.721199 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.722448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.721297 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.722448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.721728 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/907c5318-2bc1-44cb-89ef-757db10b6009-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.722637 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.722616 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.723247 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.723226 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-config\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.723380 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.723325 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-web-config\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.723476 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.723453 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/907c5318-2bc1-44cb-89ef-757db10b6009-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.724059 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.724017 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.724059 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.724039 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.724221 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.724169 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.724279 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.724237 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/907c5318-2bc1-44cb-89ef-757db10b6009-config-out\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.724704 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.724685 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.725375 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.725359 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/907c5318-2bc1-44cb-89ef-757db10b6009-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.778392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.778371 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrhw\" (UniqueName: \"kubernetes.io/projected/907c5318-2bc1-44cb-89ef-757db10b6009-kube-api-access-7mrhw\") pod \"prometheus-k8s-0\" (UID: \"907c5318-2bc1-44cb-89ef-757db10b6009\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.847597 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.847523 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:07.949322 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:07.949264 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d7f1a2-5634-44da-9108-4f1e08a232a0" path="/var/lib/kubelet/pods/10d7f1a2-5634-44da-9108-4f1e08a232a0/volumes" Oct 30 21:44:08.290317 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:08.287125 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:44:08.295257 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:44:08.295205 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907c5318_2bc1_44cb_89ef_757db10b6009.slice/crio-83246da215c29b7992e5cced36a69c50deea33320cfe7b26599832c2dba5fcf8 WatchSource:0}: Error finding container 83246da215c29b7992e5cced36a69c50deea33320cfe7b26599832c2dba5fcf8: Status 404 returned error can't find the container with id 83246da215c29b7992e5cced36a69c50deea33320cfe7b26599832c2dba5fcf8 Oct 30 21:44:08.307709 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:08.307684 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"83246da215c29b7992e5cced36a69c50deea33320cfe7b26599832c2dba5fcf8"} Oct 30 21:44:09.312464 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:09.312430 5508 generic.go:356] "Generic (PLEG): container finished" podID="907c5318-2bc1-44cb-89ef-757db10b6009" containerID="f07d45555fb28835e6cde06935570e89d006ab41e91c25b18f89e04b07aa3abc" exitCode=0 Oct 30 21:44:09.312878 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:09.312509 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerDied","Data":"f07d45555fb28835e6cde06935570e89d006ab41e91c25b18f89e04b07aa3abc"} Oct 30 21:44:10.320513 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.320481 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"5e3da1cdf9a39887f94c41aa58e4bacbc25ef16c454e93f2d637afe2b96fb6d5"} Oct 30 21:44:10.320898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.320573 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"0918ece7fecd20a1178e31c0def2fe5b6f339dc3a0f0ee9c7e28520a01604fae"} Oct 30 21:44:10.320898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.320601 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"a903336ce0b47983dd40b422c83db8a5f8ad56845760366c5264f861ed732c93"} Oct 30 21:44:10.320898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.320632 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"1759e073a33a05eec30b8f4bc2c20d0e0d7913bd177db8caf7f1daadf40db09e"} Oct 30 21:44:10.320898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.320655 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"32f2529961287aabd95964801f947538b0467915d4643294990326148c6629b6"} Oct 30 21:44:10.320898 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.320678 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"907c5318-2bc1-44cb-89ef-757db10b6009","Type":"ContainerStarted","Data":"54de52e1e6eaa8310ceac19b245dcd6010ab3f55bf9274feb49ba96fc0e0db64"} Oct 30 21:44:10.371545 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:10.371514 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:44:10.447452 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:10.447394 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.43114949 podStartE2EDuration="3.447376654s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.227164ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:10.446339078 +0000 UTC m=+60.635876765" watchObservedRunningTime="2026-10-30 21:44:10.447376654 +0000 UTC m=+60.636914330" Oct 30 21:44:12.848455 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:12.848421 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:19.055025 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:19.054990 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:19.212937 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:19.212902 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:21.081031 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:21.080998 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:34.143786 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:34.143751 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:34.158911 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:34.158886 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:34.433445 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:44:34.433373 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:44:40.370835 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:44:40.370795 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:45:09.587273 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:45:09.587180 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:10.371480 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:45:10.371446 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:45:14.392936 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:45:14.392902 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:21.427124 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:45:21.427087 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.165295 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:45:32.165258 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:32.317826 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:45:32.317795 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:36.793218 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:45:36.793178 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:40.371928 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:45:40.371883 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:46:03.244662 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:03.244624 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:03.403857 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:03.403826 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:04.432415 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:04.432379 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:04.577682 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:04.577646 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.371148 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:46:10.371110 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:46:21.781011 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:21.780975 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.297107 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:33.297064 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.515586 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:33.515551 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:33.680170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:33.680138 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:34.892584 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:34.892540 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:35.052368 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:35.052327 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:36.032762 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:36.032720 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:40.371713 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:46:40.371675 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:46:41.453745 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:41.453704 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:41.636611 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:41.636574 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:42.554884 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:42.554850 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:42.761174 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:42.761141 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:42.956099 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:42.956065 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:46.682778 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:46:46.682750 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:10.371853 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:47:10.371809 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:47:15.842223 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:15.842170 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:17.065445 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.065413 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:17.566512 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.566479 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp"] Oct 30 21:47:17.568532 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.568517 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.611209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.611186 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbvg\" (UniqueName: \"kubernetes.io/projected/dd083b66-2b77-4cf0-b166-857d5b7430ea-kube-api-access-csbvg\") pod \"jobset-operator-674cd59dc5-m9lnp\" (UID: \"dd083b66-2b77-4cf0-b166-857d5b7430ea\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.611354 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.611219 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd083b66-2b77-4cf0-b166-857d5b7430ea-tmp\") pod \"jobset-operator-674cd59dc5-m9lnp\" (UID: \"dd083b66-2b77-4cf0-b166-857d5b7430ea\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.671545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.671510 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:47:17.671545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.671518 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:47:17.671752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.671569 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-8btwv\"" Oct 30 21:47:17.672244 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.672221 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp"] Oct 30 21:47:17.711673 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.711632 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbvg\" (UniqueName: \"kubernetes.io/projected/dd083b66-2b77-4cf0-b166-857d5b7430ea-kube-api-access-csbvg\") pod \"jobset-operator-674cd59dc5-m9lnp\" (UID: \"dd083b66-2b77-4cf0-b166-857d5b7430ea\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.711796 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.711683 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd083b66-2b77-4cf0-b166-857d5b7430ea-tmp\") pod \"jobset-operator-674cd59dc5-m9lnp\" (UID: \"dd083b66-2b77-4cf0-b166-857d5b7430ea\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.712069 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.712051 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd083b66-2b77-4cf0-b166-857d5b7430ea-tmp\") pod \"jobset-operator-674cd59dc5-m9lnp\" (UID: \"dd083b66-2b77-4cf0-b166-857d5b7430ea\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.769219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.769193 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbvg\" (UniqueName: \"kubernetes.io/projected/dd083b66-2b77-4cf0-b166-857d5b7430ea-kube-api-access-csbvg\") pod \"jobset-operator-674cd59dc5-m9lnp\" (UID: \"dd083b66-2b77-4cf0-b166-857d5b7430ea\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:17.888252 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:17.888231 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" Oct 30 21:47:18.263874 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:18.263783 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp"] Oct 30 21:47:18.267594 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:47:18.267555 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd083b66_2b77_4cf0_b166_857d5b7430ea.slice/crio-ecdd0ceff0ffbdccf5b71db4adbbf24fa953d314cb260d092715e0c7df928a1c WatchSource:0}: Error finding container ecdd0ceff0ffbdccf5b71db4adbbf24fa953d314cb260d092715e0c7df928a1c: Status 404 returned error can't find the container with id ecdd0ceff0ffbdccf5b71db4adbbf24fa953d314cb260d092715e0c7df928a1c Oct 30 21:47:18.988501 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:18.988467 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" event={"ID":"dd083b66-2b77-4cf0-b166-857d5b7430ea","Type":"ContainerStarted","Data":"ecdd0ceff0ffbdccf5b71db4adbbf24fa953d314cb260d092715e0c7df928a1c"} Oct 30 21:47:21.594617 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:21.594577 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:22.001823 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:22.001737 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" event={"ID":"dd083b66-2b77-4cf0-b166-857d5b7430ea","Type":"ContainerStarted","Data":"ff80cf6b3449f5ace16c540b163f65ec0b6c0a2e9d4ce8df9a85b8dfba9ea402"} Oct 30 21:47:22.118167 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:22.118105 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-m9lnp" podStartSLOduration=1.68193021 podStartE2EDuration="5.118084587s" totalImagesPullingTime="3.436154377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:22.117138588 +0000 UTC m=+252.306676283" watchObservedRunningTime="2026-10-30 21:47:22.118084587 +0000 UTC m=+252.307622263" Oct 30 21:47:22.123105 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:22.123079 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:28.363655 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:28.363608 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:30.029900 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:30.029853 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:31.430663 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:31.430631 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:31.736176 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:31.736100 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.151767 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.151737 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.448661 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.448566 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf"] Oct 30 21:47:32.452181 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.452159 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.533342 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.533287 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthbf\" (UniqueName: \"kubernetes.io/projected/7355813a-a26d-4321-918a-c206560e85a3-kube-api-access-vthbf\") pod \"cert-manager-webhook-5dbbbf75cc-2xtqf\" (UID: \"7355813a-a26d-4321-918a-c206560e85a3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.533491 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.533392 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7355813a-a26d-4321-918a-c206560e85a3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2xtqf\" (UID: \"7355813a-a26d-4321-918a-c206560e85a3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.560019 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.559992 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:47:32.560019 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.560003 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:47:32.560186 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.560050 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nl6qb\"" Oct 30 21:47:32.560726 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.560709 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf"] Oct 30 21:47:32.634608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.634580 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthbf\" (UniqueName: \"kubernetes.io/projected/7355813a-a26d-4321-918a-c206560e85a3-kube-api-access-vthbf\") pod \"cert-manager-webhook-5dbbbf75cc-2xtqf\" (UID: \"7355813a-a26d-4321-918a-c206560e85a3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.634758 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.634609 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7355813a-a26d-4321-918a-c206560e85a3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2xtqf\" (UID: \"7355813a-a26d-4321-918a-c206560e85a3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.701187 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.701115 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthbf\" (UniqueName: \"kubernetes.io/projected/7355813a-a26d-4321-918a-c206560e85a3-kube-api-access-vthbf\") pod \"cert-manager-webhook-5dbbbf75cc-2xtqf\" (UID: \"7355813a-a26d-4321-918a-c206560e85a3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.701348 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.701282 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7355813a-a26d-4321-918a-c206560e85a3-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2xtqf\" (UID: \"7355813a-a26d-4321-918a-c206560e85a3\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.763363 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.763341 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:32.833005 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:32.832933 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:33.142035 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:33.142003 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf"] Oct 30 21:47:33.146781 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:47:33.146753 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7355813a_a26d_4321_918a_c206560e85a3.slice/crio-aa7cd7e9ba8b1ef82d68a65ebc76389c00dceb812a5cc9e5bfa9ec12777b9df0 WatchSource:0}: Error finding container aa7cd7e9ba8b1ef82d68a65ebc76389c00dceb812a5cc9e5bfa9ec12777b9df0: Status 404 returned error can't find the container with id aa7cd7e9ba8b1ef82d68a65ebc76389c00dceb812a5cc9e5bfa9ec12777b9df0 Oct 30 21:47:33.485427 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:33.485351 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:34.046965 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:34.046933 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" event={"ID":"7355813a-a26d-4321-918a-c206560e85a3","Type":"ContainerStarted","Data":"aa7cd7e9ba8b1ef82d68a65ebc76389c00dceb812a5cc9e5bfa9ec12777b9df0"} Oct 30 21:47:34.340589 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:34.340550 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.791653 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:35.791613 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:35.964706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:35.964679 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:38.001135 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.001105 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:38.065271 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.065235 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" event={"ID":"7355813a-a26d-4321-918a-c206560e85a3","Type":"ContainerStarted","Data":"d055789880e6a63d70db6977472f8374a892ea45e75de0ddb13930abafaa0375"} Oct 30 21:47:38.065437 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.065319 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:38.180280 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.180233 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" podStartSLOduration=2.267448994 podStartE2EDuration="6.180220579s" totalImagesPullingTime="3.912771585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:38.179432462 +0000 UTC m=+268.368970138" watchObservedRunningTime="2026-10-30 21:47:38.180220579 +0000 UTC m=+268.369758253" Oct 30 21:47:38.195466 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.195434 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:38.446178 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.446138 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:38.720912 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:38.720832 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.153121 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.153095 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.394155 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.394126 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g"] Oct 30 21:47:39.397627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.397604 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.473707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.473682 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.476496 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.476466 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj"] Oct 30 21:47:39.479392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.479376 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.492581 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.492560 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a68e716-6045-4683-b61a-3b3a89a8f9f1-tmp\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.492652 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.492599 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2bn6\" (UniqueName: \"kubernetes.io/projected/7a68e716-6045-4683-b61a-3b3a89a8f9f1-kube-api-access-p2bn6\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.492652 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.492641 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-metrics-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.492749 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.492731 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.492784 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.492766 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7a68e716-6045-4683-b61a-3b3a89a8f9f1-manifests\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.492813 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.492797 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.533868 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.533848 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g"] Oct 30 21:47:39.542804 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.542782 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:47:39.542940 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.542923 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:47:39.543363 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.543350 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:47:39.543467 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.543445 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:47:39.543670 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.543652 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-w7mjw\"" Oct 30 21:47:39.543801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.543733 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:47:39.543885 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.543873 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qzfd8\"" Oct 30 21:47:39.545371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.545346 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj"] Oct 30 21:47:39.593531 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593507 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.593627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593552 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7a68e716-6045-4683-b61a-3b3a89a8f9f1-manifests\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.593627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593585 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a140c51-bc2f-48b5-8b00-a558845657b2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tg6rj\" (UID: \"6a140c51-bc2f-48b5-8b00-a558845657b2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.593627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593620 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.593759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593678 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a68e716-6045-4683-b61a-3b3a89a8f9f1-tmp\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.593759 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593715 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq66d\" (UniqueName: \"kubernetes.io/projected/6a140c51-bc2f-48b5-8b00-a558845657b2-kube-api-access-zq66d\") pod \"cert-manager-cainjector-5c57bbd4fc-tg6rj\" (UID: \"6a140c51-bc2f-48b5-8b00-a558845657b2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.593862 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593793 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2bn6\" (UniqueName: \"kubernetes.io/projected/7a68e716-6045-4683-b61a-3b3a89a8f9f1-kube-api-access-p2bn6\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.593862 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593833 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-metrics-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.593966 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.593921 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7a68e716-6045-4683-b61a-3b3a89a8f9f1-manifests\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.596059 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.596032 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a68e716-6045-4683-b61a-3b3a89a8f9f1-tmp\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.596348 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.596292 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.596632 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.596614 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-metrics-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.596739 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.596722 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a68e716-6045-4683-b61a-3b3a89a8f9f1-webhook-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.654703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.654594 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:39.656500 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.656477 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2bn6\" (UniqueName: \"kubernetes.io/projected/7a68e716-6045-4683-b61a-3b3a89a8f9f1-kube-api-access-p2bn6\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-nf54g\" (UID: \"7a68e716-6045-4683-b61a-3b3a89a8f9f1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.694187 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.694167 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a140c51-bc2f-48b5-8b00-a558845657b2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tg6rj\" (UID: \"6a140c51-bc2f-48b5-8b00-a558845657b2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.694281 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.694213 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq66d\" (UniqueName: \"kubernetes.io/projected/6a140c51-bc2f-48b5-8b00-a558845657b2-kube-api-access-zq66d\") pod \"cert-manager-cainjector-5c57bbd4fc-tg6rj\" (UID: \"6a140c51-bc2f-48b5-8b00-a558845657b2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.714213 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.714189 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:39.751089 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.750942 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq66d\" (UniqueName: \"kubernetes.io/projected/6a140c51-bc2f-48b5-8b00-a558845657b2-kube-api-access-zq66d\") pod \"cert-manager-cainjector-5c57bbd4fc-tg6rj\" (UID: \"6a140c51-bc2f-48b5-8b00-a558845657b2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.751089 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.751035 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a140c51-bc2f-48b5-8b00-a558845657b2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tg6rj\" (UID: \"6a140c51-bc2f-48b5-8b00-a558845657b2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.790609 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.790582 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" Oct 30 21:47:39.843190 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:39.843074 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:40.075159 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:40.075126 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:40.205753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:40.205720 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g"] Oct 30 21:47:40.209099 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:47:40.209071 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a68e716_6045_4683_b61a_3b3a89a8f9f1.slice/crio-70019b155f81a6b536dcef62a2a4a4571bd7c6ea526689f82000b40493d12769 WatchSource:0}: Error finding container 70019b155f81a6b536dcef62a2a4a4571bd7c6ea526689f82000b40493d12769: Status 404 returned error can't find the container with id 70019b155f81a6b536dcef62a2a4a4571bd7c6ea526689f82000b40493d12769 Oct 30 21:47:40.246170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:40.246152 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj"] Oct 30 21:47:40.249593 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:47:40.249568 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a140c51_bc2f_48b5_8b00_a558845657b2.slice/crio-ad5c22e2c11ce9dc4f1a75ae2c30277450879bfa2adf62bae27de409209f7d24 WatchSource:0}: Error finding container ad5c22e2c11ce9dc4f1a75ae2c30277450879bfa2adf62bae27de409209f7d24: Status 404 returned error can't find the container with id ad5c22e2c11ce9dc4f1a75ae2c30277450879bfa2adf62bae27de409209f7d24 Oct 30 21:47:40.273222 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:40.273201 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:40.370788 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:47:40.370720 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:47:41.077369 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:41.077329 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" event={"ID":"6a140c51-bc2f-48b5-8b00-a558845657b2","Type":"ContainerStarted","Data":"cc20451894b75268eb71d2c5ebf925f4dfeb0760c10b2106887a2213ccc27e22"} Oct 30 21:47:41.077565 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:41.077381 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" event={"ID":"6a140c51-bc2f-48b5-8b00-a558845657b2","Type":"ContainerStarted","Data":"ad5c22e2c11ce9dc4f1a75ae2c30277450879bfa2adf62bae27de409209f7d24"} Oct 30 21:47:41.078738 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:41.078712 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" event={"ID":"7a68e716-6045-4683-b61a-3b3a89a8f9f1","Type":"ContainerStarted","Data":"70019b155f81a6b536dcef62a2a4a4571bd7c6ea526689f82000b40493d12769"} Oct 30 21:47:41.398675 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:41.398584 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.052018 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:42.051977 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.274660 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:42.274614 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:42.460719 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:42.460688 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.098013 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.097977 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.106356 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.106203 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tg6rj" podStartSLOduration=4.106185588 podStartE2EDuration="4.106185588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:41.192988029 +0000 UTC m=+271.382525706" watchObservedRunningTime="2026-10-30 21:47:43.106185588 +0000 UTC m=+273.295723265" Oct 30 21:47:43.106618 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.106587 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-krfmf"] Oct 30 21:47:43.110179 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.110160 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.173693 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.173672 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dkhv9\"" Oct 30 21:47:43.223869 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.223833 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-krfmf"] Oct 30 21:47:43.231295 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.231272 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84f70940-0c76-4469-8b42-1ac8d629e91e-bound-sa-token\") pod \"cert-manager-597b95cb84-krfmf\" (UID: \"84f70940-0c76-4469-8b42-1ac8d629e91e\") " pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.231441 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.231345 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zwm\" (UniqueName: \"kubernetes.io/projected/84f70940-0c76-4469-8b42-1ac8d629e91e-kube-api-access-q5zwm\") pod \"cert-manager-597b95cb84-krfmf\" (UID: \"84f70940-0c76-4469-8b42-1ac8d629e91e\") " pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.285523 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.285484 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:43.332749 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.332715 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84f70940-0c76-4469-8b42-1ac8d629e91e-bound-sa-token\") pod \"cert-manager-597b95cb84-krfmf\" (UID: \"84f70940-0c76-4469-8b42-1ac8d629e91e\") " pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.332916 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.332776 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zwm\" (UniqueName: \"kubernetes.io/projected/84f70940-0c76-4469-8b42-1ac8d629e91e-kube-api-access-q5zwm\") pod \"cert-manager-597b95cb84-krfmf\" (UID: \"84f70940-0c76-4469-8b42-1ac8d629e91e\") " pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.390289 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.390218 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zwm\" (UniqueName: \"kubernetes.io/projected/84f70940-0c76-4469-8b42-1ac8d629e91e-kube-api-access-q5zwm\") pod \"cert-manager-597b95cb84-krfmf\" (UID: \"84f70940-0c76-4469-8b42-1ac8d629e91e\") " pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.390610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.390588 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84f70940-0c76-4469-8b42-1ac8d629e91e-bound-sa-token\") pod \"cert-manager-597b95cb84-krfmf\" (UID: \"84f70940-0c76-4469-8b42-1ac8d629e91e\") " pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.424696 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.424661 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-krfmf" Oct 30 21:47:43.809475 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:43.809448 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-krfmf"] Oct 30 21:47:43.812773 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:47:43.812746 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84f70940_0c76_4469_8b42_1ac8d629e91e.slice/crio-3ccb1a43a220d7568c3dec22555ff0766318aec7a03b80ef6ecd639398355101 WatchSource:0}: Error finding container 3ccb1a43a220d7568c3dec22555ff0766318aec7a03b80ef6ecd639398355101: Status 404 returned error can't find the container with id 3ccb1a43a220d7568c3dec22555ff0766318aec7a03b80ef6ecd639398355101 Oct 30 21:47:44.012607 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.012531 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:44.071889 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.071858 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" Oct 30 21:47:44.094617 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.094583 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-krfmf" event={"ID":"84f70940-0c76-4469-8b42-1ac8d629e91e","Type":"ContainerStarted","Data":"d206d7bc5fb25f1308ff0cd034bb3e5229bdab45b832c3a88f0f5a908bef9678"} Oct 30 21:47:44.094773 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.094626 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-krfmf" event={"ID":"84f70940-0c76-4469-8b42-1ac8d629e91e","Type":"ContainerStarted","Data":"3ccb1a43a220d7568c3dec22555ff0766318aec7a03b80ef6ecd639398355101"} Oct 30 21:47:44.096247 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.096223 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" event={"ID":"7a68e716-6045-4683-b61a-3b3a89a8f9f1","Type":"ContainerStarted","Data":"2d55fd89a922815b19a9371bd155a21b3ae8860b67e93b76bbe0a9c331c6b36d"} Oct 30 21:47:44.232268 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.232240 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:44.317016 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:44.316898 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-krfmf" podStartSLOduration=1.316884528 podStartE2EDuration="1.316884528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:44.316113277 +0000 UTC m=+274.505650971" watchObservedRunningTime="2026-10-30 21:47:44.316884528 +0000 UTC m=+274.506422202" Oct 30 21:47:45.101500 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:45.101461 5508 generic.go:356] "Generic (PLEG): container finished" podID="7a68e716-6045-4683-b61a-3b3a89a8f9f1" containerID="2d55fd89a922815b19a9371bd155a21b3ae8860b67e93b76bbe0a9c331c6b36d" exitCode=0 Oct 30 21:47:45.101934 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:45.101546 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" event={"ID":"7a68e716-6045-4683-b61a-3b3a89a8f9f1","Type":"ContainerDied","Data":"2d55fd89a922815b19a9371bd155a21b3ae8860b67e93b76bbe0a9c331c6b36d"} Oct 30 21:47:46.107285 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:46.107244 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" event={"ID":"7a68e716-6045-4683-b61a-3b3a89a8f9f1","Type":"ContainerStarted","Data":"a877de4560b1e69490e75bf951f8775f8a677f8780b022335b743b28c720cb20"} Oct 30 21:47:46.107765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:46.107354 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:46.294552 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:46.294462 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" podStartSLOduration=2.612444258 podStartE2EDuration="7.294447688s" totalImagesPullingTime="4.189040212s" totalInitContainerRuntime="492.963218ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:46.244810413 +0000 UTC m=+276.434348123" watchObservedRunningTime="2026-10-30 21:47:46.294447688 +0000 UTC m=+276.483985363" Oct 30 21:47:50.571634 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:50.571556 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:50.747096 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:50.747066 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.735484 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:55.735452 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:55.902678 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:55.902649 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:56.066289 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:56.066212 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.114392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:57.114357 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-nf54g" Oct 30 21:47:57.604706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:57.604674 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:57.771842 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:47:57.771801 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:02.905292 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:02.905254 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:02.915776 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:02.915737 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz"] Oct 30 21:48:02.923818 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:02.923797 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:02.991050 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:02.991020 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:02.991205 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:02.991064 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2xw\" (UniqueName: \"kubernetes.io/projected/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-kube-api-access-6n2xw\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:02.991205 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:02.991086 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.025133 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.025095 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:48:03.025287 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.025162 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6d9db\"" Oct 30 21:48:03.025287 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.025176 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 21:48:03.025287 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.025210 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 21:48:03.025287 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.025190 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:48:03.026501 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.026479 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz"] Oct 30 21:48:03.092009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.091977 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2xw\" (UniqueName: \"kubernetes.io/projected/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-kube-api-access-6n2xw\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.092009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.092016 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.092192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.092071 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.092717 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.092684 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.094567 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.094547 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.135924 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.135892 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:03.148710 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.148688 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2xw\" (UniqueName: \"kubernetes.io/projected/f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99-kube-api-access-6n2xw\") pod \"kuadrant-console-plugin-66c4fb575d-m84rz\" (UID: \"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.235054 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.234968 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" Oct 30 21:48:03.601761 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:03.601696 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz"] Oct 30 21:48:03.605283 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:48:03.605234 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0f9640f_32ec_4f9f_8f4d_0648f1fd2f99.slice/crio-0bd0a634ec7b76d21928ac085c23c98a0144dc6c092f128c1bfbe1cfe9dc14c7 WatchSource:0}: Error finding container 0bd0a634ec7b76d21928ac085c23c98a0144dc6c092f128c1bfbe1cfe9dc14c7: Status 404 returned error can't find the container with id 0bd0a634ec7b76d21928ac085c23c98a0144dc6c092f128c1bfbe1cfe9dc14c7 Oct 30 21:48:04.135992 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:04.135956 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:04.184296 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:04.184263 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" event={"ID":"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99","Type":"ContainerStarted","Data":"0bd0a634ec7b76d21928ac085c23c98a0144dc6c092f128c1bfbe1cfe9dc14c7"} Oct 30 21:48:04.296364 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:04.296329 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:09.030687 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:09.030637 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:09.180771 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:09.180743 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:09.205941 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:09.205916 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" event={"ID":"f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99","Type":"ContainerStarted","Data":"a4a0c6860318e858bdd2f179559e5effe2a2377bd8f9bfd28a5f56a56c5fffe3"} Oct 30 21:48:09.318132 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:09.318029 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-m84rz" podStartSLOduration=1.795373074 podStartE2EDuration="7.318016692s" totalImagesPullingTime="5.522643618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:09.317641521 +0000 UTC m=+299.507179195" watchObservedRunningTime="2026-10-30 21:48:09.318016692 +0000 UTC m=+299.507554366" Oct 30 21:48:10.372379 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:48:10.372332 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:48:11.084714 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:11.084678 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.322856 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:11.322818 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:11.696601 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:11.696567 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.600366 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:14.600335 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.785221 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:14.785185 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:14.971354 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:14.971258 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:15.386572 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:15.386534 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.661060 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.661023 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.782119 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.782086 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh"] Oct 30 21:48:16.786389 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.786361 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:16.838192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.838155 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:48:16.838364 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.838345 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:48:16.888211 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.888167 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh"] Oct 30 21:48:16.902101 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.902065 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:48:16.909024 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.909000 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.918196 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.918142 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-tls-certs\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:16.918196 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.918175 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-tmp\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:16.918324 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.918269 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq72f\" (UniqueName: \"kubernetes.io/projected/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-kube-api-access-dq72f\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:16.925607 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.925585 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:48:16.925707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:16.925617 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x8f47\"" Oct 30 21:48:17.019341 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.019288 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-tls-certs\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.019341 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.019344 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-tmp\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.019527 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.019398 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq72f\" (UniqueName: \"kubernetes.io/projected/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-kube-api-access-dq72f\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.021923 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.021897 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-tmp\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.022065 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.022046 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-tls-certs\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.076370 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.076343 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq72f\" (UniqueName: \"kubernetes.io/projected/d755f93c-fd87-4cb6-8bd0-5c331e10d8eb-kube-api-access-dq72f\") pod \"kube-auth-proxy-5cbcf87c76-7h2zh\" (UID: \"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb\") " pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.097363 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.097295 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" Oct 30 21:48:17.394672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.394639 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.477117 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.477086 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh"] Oct 30 21:48:17.480777 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:48:17.480747 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd755f93c_fd87_4cb6_8bd0_5c331e10d8eb.slice/crio-57ec6ac303132488a19a2873aacac079f95d96a55e2971b05a053e6a513f4f36 WatchSource:0}: Error finding container 57ec6ac303132488a19a2873aacac079f95d96a55e2971b05a053e6a513f4f36: Status 404 returned error can't find the container with id 57ec6ac303132488a19a2873aacac079f95d96a55e2971b05a053e6a513f4f36 Oct 30 21:48:17.482693 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.482672 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:48:17.550703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.550666 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.712026 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.711954 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:17.867295 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:17.867267 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:18.245489 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:18.245459 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" event={"ID":"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb","Type":"ContainerStarted","Data":"57ec6ac303132488a19a2873aacac079f95d96a55e2971b05a053e6a513f4f36"} Oct 30 21:48:19.200182 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:19.200146 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:19.360402 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:19.360367 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:19.532760 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:19.532674 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:19.865438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:19.865391 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:20.110569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.110531 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:20.234336 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.234288 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t"] Oct 30 21:48:20.237469 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.237452 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.256926 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.256898 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" event={"ID":"d755f93c-fd87-4cb6-8bd0-5c331e10d8eb","Type":"ContainerStarted","Data":"946af77b45153797e64c4b69f4b0448d7026046e94110b74df31d90e7cc189b6"} Oct 30 21:48:20.289034 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.289008 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:48:20.338289 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.338260 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:48:20.338427 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.338412 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:48:20.338478 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.338458 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-vh9p7\"" Oct 30 21:48:20.339448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.339430 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t"] Oct 30 21:48:20.350392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.350361 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8caa4b10-97d0-4cb9-a725-f327a943bbc8-cert\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.350494 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.350477 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsxk\" (UniqueName: \"kubernetes.io/projected/8caa4b10-97d0-4cb9-a725-f327a943bbc8-kube-api-access-4tsxk\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.350535 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.350505 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caa4b10-97d0-4cb9-a725-f327a943bbc8-metrics-certs\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.350585 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.350533 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8caa4b10-97d0-4cb9-a725-f327a943bbc8-manager-config\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.451131 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.451054 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsxk\" (UniqueName: \"kubernetes.io/projected/8caa4b10-97d0-4cb9-a725-f327a943bbc8-kube-api-access-4tsxk\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.451131 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.451091 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caa4b10-97d0-4cb9-a725-f327a943bbc8-metrics-certs\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.451131 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.451118 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8caa4b10-97d0-4cb9-a725-f327a943bbc8-manager-config\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.451421 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.451209 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8caa4b10-97d0-4cb9-a725-f327a943bbc8-cert\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.451924 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.451901 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8caa4b10-97d0-4cb9-a725-f327a943bbc8-manager-config\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.453698 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.453677 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caa4b10-97d0-4cb9-a725-f327a943bbc8-metrics-certs\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.453919 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.453894 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8caa4b10-97d0-4cb9-a725-f327a943bbc8-cert\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.463693 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.463635 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cbcf87c76-7h2zh" podStartSLOduration=1.759930505 podStartE2EDuration="4.463617628s" totalImagesPullingTime="2.703687123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:20.461768931 +0000 UTC m=+310.651306630" watchObservedRunningTime="2026-10-30 21:48:20.463617628 +0000 UTC m=+310.653155304" Oct 30 21:48:20.510186 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.510159 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsxk\" (UniqueName: \"kubernetes.io/projected/8caa4b10-97d0-4cb9-a725-f327a943bbc8-kube-api-access-4tsxk\") pod \"jobset-controller-manager-7b5964b65b-qn69t\" (UID: \"8caa4b10-97d0-4cb9-a725-f327a943bbc8\") " pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.549032 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.549000 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:20.922765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:20.922732 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t"] Oct 30 21:48:20.925696 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:48:20.925668 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caa4b10_97d0_4cb9_a725_f327a943bbc8.slice/crio-156aef0474fca4943583ed86914ffa6ea2621e5b1cc76d99a76d9f0376820f70 WatchSource:0}: Error finding container 156aef0474fca4943583ed86914ffa6ea2621e5b1cc76d99a76d9f0376820f70: Status 404 returned error can't find the container with id 156aef0474fca4943583ed86914ffa6ea2621e5b1cc76d99a76d9f0376820f70 Oct 30 21:48:21.260687 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:21.260608 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:21.263070 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:21.263042 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" event={"ID":"8caa4b10-97d0-4cb9-a725-f327a943bbc8","Type":"ContainerStarted","Data":"156aef0474fca4943583ed86914ffa6ea2621e5b1cc76d99a76d9f0376820f70"} Oct 30 21:48:21.413206 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:21.413171 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:22.445829 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:22.445791 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:24.742613 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:24.742574 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:24.894855 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:24.894817 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:25.059944 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:25.059830 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.287057 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:26.287023 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" event={"ID":"8caa4b10-97d0-4cb9-a725-f327a943bbc8","Type":"ContainerStarted","Data":"b31689b43d1589b2780781ca65175c88b29a6c74665fd4ca964af1c19a9ac4c8"} Oct 30 21:48:26.287562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:26.287140 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:26.400407 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:26.400346 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" podStartSLOduration=2.056143933 podStartE2EDuration="6.400327408s" totalImagesPullingTime="4.344183475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:26.399134344 +0000 UTC m=+316.588672039" watchObservedRunningTime="2026-10-30 21:48:26.400327408 +0000 UTC m=+316.589865080" Oct 30 21:48:37.297401 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:37.297373 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7b5964b65b-qn69t" Oct 30 21:48:40.371756 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:48:40.371712 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:48:42.357659 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:42.357623 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:42.555088 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:42.555058 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:43.052616 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:48:43.052575 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.678839 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:03.678808 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.842168 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:03.842134 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.999504 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:03.999417 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.131940 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:04.131906 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:04.285233 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:04.285157 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.334640 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:08.334602 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:08.504604 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:08.504571 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:10.370712 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:49:10.370677 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:49:11.423466 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:11.423436 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:11.653488 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:11.653459 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:22.819277 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:22.819240 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.370850 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:49:40.370813 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:49:41.748220 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:41.748180 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.099003 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:42.098971 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.937585 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:43.937551 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:44.203746 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:44.203674 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:44.392264 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:44.392235 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.696855 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:51.696818 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.855896 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:51.855857 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.018862 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:52.018791 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.201451 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:52.201408 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.374107 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:52.374060 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.560355 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:52.560288 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.724231 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:52.724144 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:53.969248 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:53.969210 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.771379 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:54.771348 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.965609 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:54.965564 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.152396 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:59.152358 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.322362 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:59.322331 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.525108 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:59.525014 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.709941 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:59.709897 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.934331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:49:59.934263 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.109510 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:50:00.109475 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.312577 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:50:00.312486 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.488604 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:50:00.488569 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:10.371472 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:50:10.371437 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:50:23.134631 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:50:23.134586 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:40.372056 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:50:40.372018 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:51:10.371081 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:51:10.370997 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:51:17.632621 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:17.632580 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:23.524191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:23.524156 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.119254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:34.119215 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.324286 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:34.324246 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.372989 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:51:40.372948 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:51:52.550417 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:52.550375 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:52.715244 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:52.715194 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:53.315627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:53.315594 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.575703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:51:57.575662 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.631636 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:00.631604 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:04.499772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.499742 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld"] Oct 30 21:52:04.503205 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.503184 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.555428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.555395 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:52:04.604425 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.604395 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:52:04.604600 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.604446 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 21:52:04.604600 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.604499 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rghth\"" Oct 30 21:52:04.605431 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.605411 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld"] Oct 30 21:52:04.628840 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.628813 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:04.664920 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.664896 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxxz\" (UniqueName: \"kubernetes.io/projected/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-kube-api-access-snxxz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.665014 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.664933 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.665053 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.665010 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.665053 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.665028 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.665119 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.665057 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.665119 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.665074 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.765743 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.765658 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.765743 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.765708 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766056 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.765766 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxxz\" (UniqueName: \"kubernetes.io/projected/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-kube-api-access-snxxz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766056 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.765901 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766056 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.765981 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766056 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.766011 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766056 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.766052 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766246 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.766229 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.766280 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.766264 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.768295 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.768278 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.768362 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.768284 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:04.822678 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:04.822648 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxxz\" (UniqueName: \"kubernetes.io/projected/81e8a72c-35ee-4b3d-8466-6aa5adec13ab-kube-api-access-snxxz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld\" (UID: \"81e8a72c-35ee-4b3d-8466-6aa5adec13ab\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:05.115751 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:05.115721 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:05.486145 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:05.486076 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld"] Oct 30 21:52:05.489335 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:52:05.489282 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e8a72c_35ee_4b3d_8466_6aa5adec13ab.slice/crio-70c14101fc2a9ec84e564ffc8700eb65e3f39ab94cb710ac863858617d511b09 WatchSource:0}: Error finding container 70c14101fc2a9ec84e564ffc8700eb65e3f39ab94cb710ac863858617d511b09: Status 404 returned error can't find the container with id 70c14101fc2a9ec84e564ffc8700eb65e3f39ab94cb710ac863858617d511b09 Oct 30 21:52:06.190537 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:06.190501 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" event={"ID":"81e8a72c-35ee-4b3d-8466-6aa5adec13ab","Type":"ContainerStarted","Data":"70c14101fc2a9ec84e564ffc8700eb65e3f39ab94cb710ac863858617d511b09"} Oct 30 21:52:07.195873 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:07.195839 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" event={"ID":"81e8a72c-35ee-4b3d-8466-6aa5adec13ab","Type":"ContainerStarted","Data":"2b56e9526bd32c6349470f575783f1bcd143042d6fd34d24caee0d93d6f09ffa"} Oct 30 21:52:07.311220 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:07.311121 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" podStartSLOduration=1.829779456 podStartE2EDuration="3.311106616s" totalImagesPullingTime="1.48132716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:07.310958176 +0000 UTC m=+537.500495863" watchObservedRunningTime="2026-10-30 21:52:07.311106616 +0000 UTC m=+537.500644285" Oct 30 21:52:08.116490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:08.116455 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:10.371953 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:52:10.371856 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:52:10.385784 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.385758 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572"] Oct 30 21:52:10.388885 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.388868 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.466020 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.465985 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:52:10.492017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.491976 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572"] Oct 30 21:52:10.492753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.492727 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:10.518595 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.518571 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.518728 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.518602 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.518728 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.518724 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.518803 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.518743 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jnqf\" (UniqueName: \"kubernetes.io/projected/7663bb39-fca3-4911-9af6-fc39c5ffa391-kube-api-access-9jnqf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.518803 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.518783 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.518863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.518849 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7663bb39-fca3-4911-9af6-fc39c5ffa391-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619411 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619380 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619411 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619412 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jnqf\" (UniqueName: \"kubernetes.io/projected/7663bb39-fca3-4911-9af6-fc39c5ffa391-kube-api-access-9jnqf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619575 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619442 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619575 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619466 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7663bb39-fca3-4911-9af6-fc39c5ffa391-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619575 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619488 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619699 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619589 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619857 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619832 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619903 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619868 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.619983 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.619966 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.621888 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.621863 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7663bb39-fca3-4911-9af6-fc39c5ffa391-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.622089 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.622051 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7663bb39-fca3-4911-9af6-fc39c5ffa391-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.676185 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.676159 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jnqf\" (UniqueName: \"kubernetes.io/projected/7663bb39-fca3-4911-9af6-fc39c5ffa391-kube-api-access-9jnqf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g2572\" (UID: \"7663bb39-fca3-4911-9af6-fc39c5ffa391\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:10.701067 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:10.701040 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:11.072945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:11.072915 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572"] Oct 30 21:52:11.075966 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:52:11.075941 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7663bb39_fca3_4911_9af6_fc39c5ffa391.slice/crio-05c36f5005b71113c10cb66df8c4834624945e12a3120badf9f0c2075cde4c8e WatchSource:0}: Error finding container 05c36f5005b71113c10cb66df8c4834624945e12a3120badf9f0c2075cde4c8e: Status 404 returned error can't find the container with id 05c36f5005b71113c10cb66df8c4834624945e12a3120badf9f0c2075cde4c8e Oct 30 21:52:11.213880 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:11.213843 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" event={"ID":"7663bb39-fca3-4911-9af6-fc39c5ffa391","Type":"ContainerStarted","Data":"05c36f5005b71113c10cb66df8c4834624945e12a3120badf9f0c2075cde4c8e"} Oct 30 21:52:12.220146 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:12.220108 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" event={"ID":"7663bb39-fca3-4911-9af6-fc39c5ffa391","Type":"ContainerStarted","Data":"246f62742e4b6c125dcf9e2e767f39e7f2a7f014e04510e341ec3c1ce67024d8"} Oct 30 21:52:12.333942 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:12.333889 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" podStartSLOduration=1.755786304 podStartE2EDuration="2.333875239s" totalImagesPullingTime="578.088935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:12.333166854 +0000 UTC m=+542.522704529" watchObservedRunningTime="2026-10-30 21:52:12.333875239 +0000 UTC m=+542.523412916" Oct 30 21:52:12.701760 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:12.701727 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:14.425936 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.425900 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r"] Oct 30 21:52:14.429472 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.429451 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.452101 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.452075 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6cl\" (UniqueName: \"kubernetes.io/projected/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-kube-api-access-fv6cl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.452208 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.452111 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.452208 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.452183 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.452293 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.452209 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.452293 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.452227 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.452293 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.452246 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.530343 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.530291 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:52:14.531547 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.531524 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r"] Oct 30 21:52:14.543254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.543227 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:14.553448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553423 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553551 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553468 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553551 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553493 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553658 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553604 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553658 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553647 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553782 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6cl\" (UniqueName: \"kubernetes.io/projected/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-kube-api-access-fv6cl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553893 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553876 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.553980 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.553957 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.554113 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.554090 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.555981 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.555964 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.556639 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.556619 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:14.751393 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:14.751324 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6cl\" (UniqueName: \"kubernetes.io/projected/b1b6787b-0e4c-4564-a9f0-5e84ea279f38-kube-api-access-fv6cl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r\" (UID: \"b1b6787b-0e4c-4564-a9f0-5e84ea279f38\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:15.040833 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:15.040739 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:15.116381 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:15.116342 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:15.129067 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:15.129042 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:15.244471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:15.244447 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld" Oct 30 21:52:15.412380 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:15.412265 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r"] Oct 30 21:52:15.415430 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:52:15.415400 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1b6787b_0e4c_4564_a9f0_5e84ea279f38.slice/crio-33223b2938c2f015e8e3db16a645ee9a71db64532ae3c5f93cd9af9f179d612c WatchSource:0}: Error finding container 33223b2938c2f015e8e3db16a645ee9a71db64532ae3c5f93cd9af9f179d612c: Status 404 returned error can't find the container with id 33223b2938c2f015e8e3db16a645ee9a71db64532ae3c5f93cd9af9f179d612c Oct 30 21:52:16.239218 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:16.239180 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" event={"ID":"b1b6787b-0e4c-4564-a9f0-5e84ea279f38","Type":"ContainerStarted","Data":"d9ad0f85d4fe20b83aaa993afa7cf609be143e626462860785896019a63249a7"} Oct 30 21:52:16.239633 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:16.239229 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" event={"ID":"b1b6787b-0e4c-4564-a9f0-5e84ea279f38","Type":"ContainerStarted","Data":"33223b2938c2f015e8e3db16a645ee9a71db64532ae3c5f93cd9af9f179d612c"} Oct 30 21:52:16.354533 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:16.354479 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podStartSLOduration=1.769861138 podStartE2EDuration="2.35446297s" totalImagesPullingTime="584.601832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:16.353262703 +0000 UTC m=+546.542800395" watchObservedRunningTime="2026-10-30 21:52:16.35446297 +0000 UTC m=+546.544000636" Oct 30 21:52:17.041170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:17.041138 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:17.854658 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:17.854626 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:20.701395 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:20.701366 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:20.714019 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:20.713995 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:21.269868 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.269833 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g2572" Oct 30 21:52:21.711124 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.711088 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl"] Oct 30 21:52:21.714457 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.714430 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.767945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.767916 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 21:52:21.814528 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.814495 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:21.818080 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.818052 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl"] Oct 30 21:52:21.819741 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.819719 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ef4e87-d54c-4200-b643-3f1bcdce306c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.819796 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.819766 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.819860 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.819846 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.819895 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.819875 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4tv\" (UniqueName: \"kubernetes.io/projected/26ef4e87-d54c-4200-b643-3f1bcdce306c-kube-api-access-nw4tv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.819934 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.819895 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.820042 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.820025 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.920579 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.920544 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.920734 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.920600 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.920734 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.920629 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4tv\" (UniqueName: \"kubernetes.io/projected/26ef4e87-d54c-4200-b643-3f1bcdce306c-kube-api-access-nw4tv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.920734 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.920657 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.920874 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.920724 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.920874 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.920790 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ef4e87-d54c-4200-b643-3f1bcdce306c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.921058 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.921037 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.921123 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.921107 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.921195 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.921175 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.922907 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.922883 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26ef4e87-d54c-4200-b643-3f1bcdce306c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.923187 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.923171 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26ef4e87-d54c-4200-b643-3f1bcdce306c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:21.979839 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:21.979776 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4tv\" (UniqueName: \"kubernetes.io/projected/26ef4e87-d54c-4200-b643-3f1bcdce306c-kube-api-access-nw4tv\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zr9gl\" (UID: \"26ef4e87-d54c-4200-b643-3f1bcdce306c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:22.027881 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:22.027845 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:22.411124 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:22.411093 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl"] Oct 30 21:52:22.414585 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:52:22.414553 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ef4e87_d54c_4200_b643_3f1bcdce306c.slice/crio-1e28f922d13c294fbb263f5647da6c17f30688d8de3173c060c9983577b315e0 WatchSource:0}: Error finding container 1e28f922d13c294fbb263f5647da6c17f30688d8de3173c060c9983577b315e0: Status 404 returned error can't find the container with id 1e28f922d13c294fbb263f5647da6c17f30688d8de3173c060c9983577b315e0 Oct 30 21:52:23.269394 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:23.269292 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" event={"ID":"26ef4e87-d54c-4200-b643-3f1bcdce306c","Type":"ContainerStarted","Data":"bdd4cb7fd7a6de68d2e10a00db0a70ea2888c7452fcc20bf79be71eaead034e2"} Oct 30 21:52:23.269752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:23.269392 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" event={"ID":"26ef4e87-d54c-4200-b643-3f1bcdce306c","Type":"ContainerStarted","Data":"1e28f922d13c294fbb263f5647da6c17f30688d8de3173c060c9983577b315e0"} Oct 30 21:52:23.396424 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:23.396378 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" podStartSLOduration=1.800776791 podStartE2EDuration="2.396365086s" totalImagesPullingTime="595.588295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:23.394632178 +0000 UTC m=+553.584169862" watchObservedRunningTime="2026-10-30 21:52:23.396365086 +0000 UTC m=+553.585902801" Oct 30 21:52:23.865232 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:23.865195 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:24.028146 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:24.028114 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:25.041673 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:25.041634 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:25.054100 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:25.054078 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:25.290060 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:25.290033 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 21:52:29.526444 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:29.526412 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:29.700692 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:29.700664 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:30.280406 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.280376 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm"] Oct 30 21:52:30.283855 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.283840 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.340347 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.340323 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v2z7t\"" Oct 30 21:52:30.389590 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.389570 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:52:30.401859 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.401835 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.401966 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.401867 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.401966 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.401906 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.402045 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.402014 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.402081 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.402056 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.402132 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.402116 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.402175 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.402164 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4fj\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.402209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.402196 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.402241 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.402211 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.408855 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.408835 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm"] Oct 30 21:52:30.502655 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502629 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4fj\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.502802 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502669 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.502802 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502686 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.502802 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502708 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.502802 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502729 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.502802 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502766 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503054 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502827 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503054 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502870 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503054 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.502900 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503205 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.503106 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503263 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.503251 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503383 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.503361 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503481 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.503464 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.503559 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.503540 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.505220 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.505188 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.505492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.505475 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.577620 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.577588 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.606268 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.606239 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4fj\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.898502 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.898424 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:30.962118 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:30.962091 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:31.331269 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:31.331237 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm"] Oct 30 21:52:31.334035 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:52:31.334000 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05633acf_5645_4b82_81aa_c73149763847.slice/crio-f0da7f89278b3bfb89c21fa7a07f9a0e0d4276e1093405465c529cf032dfc6e1 WatchSource:0}: Error finding container f0da7f89278b3bfb89c21fa7a07f9a0e0d4276e1093405465c529cf032dfc6e1: Status 404 returned error can't find the container with id f0da7f89278b3bfb89c21fa7a07f9a0e0d4276e1093405465c529cf032dfc6e1 Oct 30 21:52:32.028452 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:32.028421 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:32.040376 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:32.040354 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:32.311229 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:32.311140 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" event={"ID":"05633acf-5645-4b82-81aa-c73149763847","Type":"ContainerStarted","Data":"f0da7f89278b3bfb89c21fa7a07f9a0e0d4276e1093405465c529cf032dfc6e1"} Oct 30 21:52:32.322674 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:32.322646 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zr9gl" Oct 30 21:52:35.334026 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:35.333994 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:35.334260 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:35.334071 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:35.334260 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:35.334100 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:36.331904 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:36.331874 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" event={"ID":"05633acf-5645-4b82-81aa-c73149763847","Type":"ContainerStarted","Data":"eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b"} Oct 30 21:52:36.500718 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:36.500652 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podStartSLOduration=2.503267911 podStartE2EDuration="6.500631527s" totalImagesPullingTime="3.997363616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:36.500166555 +0000 UTC m=+566.689704234" watchObservedRunningTime="2026-10-30 21:52:36.500631527 +0000 UTC m=+566.690169202" Oct 30 21:52:36.899511 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:36.899472 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:36.901068 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:36.901040 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:52:36.901177 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:36.901093 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 21:52:37.899691 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:37.899651 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:52:37.900174 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:37.899707 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 21:52:38.899720 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:38.899681 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:52:38.900107 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:38.899743 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 21:52:39.899666 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:39.899631 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:52:39.899866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:39.899784 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 21:52:40.371581 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:52:40.371543 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:52:40.899746 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:40.899709 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:52:40.899973 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:40.899786 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 21:52:41.898801 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:41.898762 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Oct 30 21:52:41.898970 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:41.898840 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Oct 30 21:52:43.495981 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:43.495799 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:43.496710 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:43.496190 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:43.497274 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:43.497243 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:52:43.619500 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:43.619451 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:43.837652 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:43.837619 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:45.379501 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.379470 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs"] Oct 30 21:52:45.384511 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.384495 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.491002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.490496 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs"] Oct 30 21:52:45.568221 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568184 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568221 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568220 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568241 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568339 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568388 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568480 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568439 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568634 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568545 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c48w5\" (UniqueName: \"kubernetes.io/projected/b5cf4322-153f-4647-b6f2-3e08ca5285f5-kube-api-access-c48w5\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568634 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568577 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.568634 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.568603 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.669764 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669694 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.669764 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669724 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.669764 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669742 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.669764 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669759 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669783 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669849 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669925 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c48w5\" (UniqueName: \"kubernetes.io/projected/b5cf4322-153f-4647-b6f2-3e08ca5285f5-kube-api-access-c48w5\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669950 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.669984 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670357 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.670166 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670445 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.670420 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670529 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.670505 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670651 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.670625 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.670830 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.670807 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.672314 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.672276 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.672523 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.672506 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.726206 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.726184 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cf4322-153f-4647-b6f2-3e08ca5285f5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.726682 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.726658 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c48w5\" (UniqueName: \"kubernetes.io/projected/b5cf4322-153f-4647-b6f2-3e08ca5285f5-kube-api-access-c48w5\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs\" (UID: \"b5cf4322-153f-4647-b6f2-3e08ca5285f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:45.997155 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:45.997083 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:46.422440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:46.422411 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs"] Oct 30 21:52:46.425550 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:52:46.425522 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cf4322_153f_4647_b6f2_3e08ca5285f5.slice/crio-f441a0626f9c2f4c6cd15e3bd55a7df76a74b8f8c0d5e1a1bde94947db5f3e53 WatchSource:0}: Error finding container f441a0626f9c2f4c6cd15e3bd55a7df76a74b8f8c0d5e1a1bde94947db5f3e53: Status 404 returned error can't find the container with id f441a0626f9c2f4c6cd15e3bd55a7df76a74b8f8c0d5e1a1bde94947db5f3e53 Oct 30 21:52:46.427707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:46.427673 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:46.427781 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:46.427739 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:46.427781 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:46.427767 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:52:47.402874 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:47.402831 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" event={"ID":"b5cf4322-153f-4647-b6f2-3e08ca5285f5","Type":"ContainerStarted","Data":"11fb51b25408cdbba35c56d2e08125b850a4df894b1bab5134f139d474901ad6"} Oct 30 21:52:47.403330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:47.403193 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" event={"ID":"b5cf4322-153f-4647-b6f2-3e08ca5285f5","Type":"ContainerStarted","Data":"f441a0626f9c2f4c6cd15e3bd55a7df76a74b8f8c0d5e1a1bde94947db5f3e53"} Oct 30 21:52:47.620277 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:47.620210 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podStartSLOduration=2.620190656 podStartE2EDuration="2.620190656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:47.569435991 +0000 UTC m=+577.758973688" watchObservedRunningTime="2026-10-30 21:52:47.620190656 +0000 UTC m=+577.809728333" Oct 30 21:52:47.999334 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:47.999024 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:47.999334 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:47.999131 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:52:47.999334 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:47.999174 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 21:52:48.998329 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:48.997680 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:52:48.998329 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:48.997741 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 21:52:49.998331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:49.997613 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:52:49.998331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:49.997668 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 21:52:50.999788 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:50.999740 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:52:51.000280 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:50.999815 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 21:52:51.998331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:51.997652 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:52:51.998331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:51.997730 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 21:52:52.997440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:52.997396 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:52:52.997917 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:52.997453 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 21:52:53.061387 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:53.061354 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:53.290526 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:53.290430 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:54.673217 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:54.673187 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:54.673698 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:54.673561 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:52:54.674409 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:52:54.674383 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" Oct 30 21:53:02.423991 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.423952 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg"] Oct 30 21:53:02.427507 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.427490 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.479610 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.479584 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:53:02.528744 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.528706 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:53:02.528945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.528933 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:53:02.529019 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.528997 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:53:02.529164 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.529154 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gt9r2\"" Oct 30 21:53:02.530437 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.530415 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg"] Oct 30 21:53:02.544598 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.544568 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44828d8-ee41-4682-93f9-726b3560a5f6-metrics-certs\") pod \"ai-gateway-controller-6b859fd7fd-z5xbg\" (UID: \"a44828d8-ee41-4682-93f9-726b3560a5f6\") " pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.544704 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.544641 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7bjw\" (UniqueName: \"kubernetes.io/projected/a44828d8-ee41-4682-93f9-726b3560a5f6-kube-api-access-q7bjw\") pod \"ai-gateway-controller-6b859fd7fd-z5xbg\" (UID: \"a44828d8-ee41-4682-93f9-726b3560a5f6\") " pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.646016 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.645974 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44828d8-ee41-4682-93f9-726b3560a5f6-metrics-certs\") pod \"ai-gateway-controller-6b859fd7fd-z5xbg\" (UID: \"a44828d8-ee41-4682-93f9-726b3560a5f6\") " pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.646225 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.646057 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7bjw\" (UniqueName: \"kubernetes.io/projected/a44828d8-ee41-4682-93f9-726b3560a5f6-kube-api-access-q7bjw\") pod \"ai-gateway-controller-6b859fd7fd-z5xbg\" (UID: \"a44828d8-ee41-4682-93f9-726b3560a5f6\") " pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.648640 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.648619 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44828d8-ee41-4682-93f9-726b3560a5f6-metrics-certs\") pod \"ai-gateway-controller-6b859fd7fd-z5xbg\" (UID: \"a44828d8-ee41-4682-93f9-726b3560a5f6\") " pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.702827 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.702760 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7bjw\" (UniqueName: \"kubernetes.io/projected/a44828d8-ee41-4682-93f9-726b3560a5f6-kube-api-access-q7bjw\") pod \"ai-gateway-controller-6b859fd7fd-z5xbg\" (UID: \"a44828d8-ee41-4682-93f9-726b3560a5f6\") " pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:02.740765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:02.740730 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:03.112170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:03.112138 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg"] Oct 30 21:53:03.115210 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:53:03.115180 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44828d8_ee41_4682_93f9_726b3560a5f6.slice/crio-c71eaa3ac54ea6602b5a5adaf250c5e4c67f903a1ef744bce1b128500af82949 WatchSource:0}: Error finding container c71eaa3ac54ea6602b5a5adaf250c5e4c67f903a1ef744bce1b128500af82949: Status 404 returned error can't find the container with id c71eaa3ac54ea6602b5a5adaf250c5e4c67f903a1ef744bce1b128500af82949 Oct 30 21:53:03.481666 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:03.481577 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" event={"ID":"a44828d8-ee41-4682-93f9-726b3560a5f6","Type":"ContainerStarted","Data":"c71eaa3ac54ea6602b5a5adaf250c5e4c67f903a1ef744bce1b128500af82949"} Oct 30 21:53:06.499092 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:06.499055 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" event={"ID":"a44828d8-ee41-4682-93f9-726b3560a5f6","Type":"ContainerStarted","Data":"56de547eff89ab1744da7e91104e360ee498bebad60725bd24c05ef1b181d119"} Oct 30 21:53:06.499526 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:06.499180 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:06.618039 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:06.617982 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" podStartSLOduration=1.8260642790000001 podStartE2EDuration="4.617967628s" totalImagesPullingTime="2.791903349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:06.617027908 +0000 UTC m=+596.806565585" watchObservedRunningTime="2026-10-30 21:53:06.617967628 +0000 UTC m=+596.807505300" Oct 30 21:53:10.372369 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:53:10.372332 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:53:17.505820 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:17.505788 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6b859fd7fd-z5xbg" Oct 30 21:53:19.336187 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:19.336153 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:19.799782 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:19.799692 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:21.384293 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:21.384255 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:21.670691 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:21.670588 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:23.284439 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:23.284406 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:23.640529 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:23.640494 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.408000 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:24.407967 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:40.371818 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:53:40.371760 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:53:48.802538 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:48.802449 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:48.963611 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:48.963578 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.539224 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.539190 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.723544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.723516 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc"] Oct 30 21:53:49.727752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.727736 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.783851 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.783817 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.783992 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.783875 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.783992 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.783970 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9vt\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.784066 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.784000 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.784099 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.784084 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.784153 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.784137 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.784190 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.784179 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.784221 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.784210 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.784261 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.784250 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.794095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.794034 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc"] Oct 30 21:53:49.828548 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.828524 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b0c55f-openshift-default-dockercfg-gqvhz\"" Oct 30 21:53:49.832490 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.832467 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.885328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885282 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9vt\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885335 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885377 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885404 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885441 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885645 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885464 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885645 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885490 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885645 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885559 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885645 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885603 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885857 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885771 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.885913 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.885892 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.886044 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.886024 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.886219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.886195 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.886219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.886212 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.888100 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.888071 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.888293 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.888276 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.942623 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.942599 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9vt\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.944582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.944559 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:49.993055 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:49.993027 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.039721 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.039686 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:50.179552 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.179515 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.358355 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.358320 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.372148 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.372119 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt"] Oct 30 21:53:50.375386 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.375368 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.433125 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.433092 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-29cbad-openshift-default-dockercfg-c5s8g\"" Oct 30 21:53:50.439405 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.439373 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt"] Oct 30 21:53:50.492478 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492447 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492481 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492627 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492603 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492698 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492644 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492736 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492713 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbnr\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492770 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492744 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492804 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492781 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492854 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492841 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.492888 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.492877 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.519543 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.519510 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc"] Oct 30 21:53:50.523617 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:53:50.523593 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73eb82d_1732_4b29_b53a_104b03d75e32.slice/crio-cebe6d6d9a76a11646a86c89884253b27381c672a6281c23fc52e5596c844b40 WatchSource:0}: Error finding container cebe6d6d9a76a11646a86c89884253b27381c672a6281c23fc52e5596c844b40: Status 404 returned error can't find the container with id cebe6d6d9a76a11646a86c89884253b27381c672a6281c23fc52e5596c844b40 Oct 30 21:53:50.525722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.525695 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:50.525774 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.525759 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:50.525810 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.525788 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:50.542715 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.542692 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.593662 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593639 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593670 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593707 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593740 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593961 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593776 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593961 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593852 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593961 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593877 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593961 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593905 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbnr\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.593961 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.593942 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.594477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.594457 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.594587 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.594520 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.594587 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.594531 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.594706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.594650 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.594762 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.594698 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.596178 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.596160 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.596425 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.596410 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.650999 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.650928 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.651235 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.651212 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbnr\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.687358 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.687332 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:50.692784 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.692760 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" event={"ID":"d73eb82d-1732-4b29-b53a-104b03d75e32","Type":"ContainerStarted","Data":"559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85"} Oct 30 21:53:50.692905 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.692814 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" event={"ID":"d73eb82d-1732-4b29-b53a-104b03d75e32","Type":"ContainerStarted","Data":"cebe6d6d9a76a11646a86c89884253b27381c672a6281c23fc52e5596c844b40"} Oct 30 21:53:50.726861 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.726825 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:50.823587 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.823533 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podStartSLOduration=1.823519035 podStartE2EDuration="1.823519035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:50.822245387 +0000 UTC m=+641.011783091" watchObservedRunningTime="2026-10-30 21:53:50.823519035 +0000 UTC m=+641.013056704" Oct 30 21:53:50.900431 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:50.900392 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.040227 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.040158 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:51.045140 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.045114 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:51.151417 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.151384 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.165679 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.165627 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt"] Oct 30 21:53:51.168369 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:53:51.168336 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c99a268_d806_4b8f_8aee_824e3a6d6b1a.slice/crio-0f06be8954e088b68b89795deda7387202eff3cbe75d9c39188aecd6cccce202 WatchSource:0}: Error finding container 0f06be8954e088b68b89795deda7387202eff3cbe75d9c39188aecd6cccce202: Status 404 returned error can't find the container with id 0f06be8954e088b68b89795deda7387202eff3cbe75d9c39188aecd6cccce202 Oct 30 21:53:51.171055 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.171022 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:51.171170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.171098 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:51.171170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.171149 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:51.329113 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.329083 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.407401 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.407370 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9"] Oct 30 21:53:51.410773 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.410756 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.462588 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.462561 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-dde1e8-openshift-default-dockercfg-5l6vx\"" Oct 30 21:53:51.481335 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.481284 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9"] Oct 30 21:53:51.501051 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.500965 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.502764 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.502735 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.502879 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.502779 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.502879 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.502826 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.503004 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.502913 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.503004 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.502970 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.503131 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.503105 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.503190 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.503174 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcz22\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.503281 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.503259 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.503364 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.503293 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604434 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604362 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604434 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604416 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcz22\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604480 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604504 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604535 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604563 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604611 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604641 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.604919 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.604692 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.605038 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.605010 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.605106 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.605082 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.605154 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.605127 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.605198 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.605184 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.605276 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.605260 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.607499 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.607480 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.607823 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.607801 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.661124 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.661094 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.661282 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.661248 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcz22\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.673292 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.673264 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.699088 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.699056 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" event={"ID":"0c99a268-d806-4b8f-8aee-824e3a6d6b1a","Type":"ContainerStarted","Data":"8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807"} Oct 30 21:53:51.699202 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.699154 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" event={"ID":"0c99a268-d806-4b8f-8aee-824e3a6d6b1a","Type":"ContainerStarted","Data":"0f06be8954e088b68b89795deda7387202eff3cbe75d9c39188aecd6cccce202"} Oct 30 21:53:51.699202 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.699192 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:51.700008 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.699991 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:53:51.723992 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.723969 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:51.832980 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.832923 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podStartSLOduration=1.832905388 podStartE2EDuration="1.832905388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.831987934 +0000 UTC m=+642.021525660" watchObservedRunningTime="2026-10-30 21:53:51.832905388 +0000 UTC m=+642.022443062" Oct 30 21:53:51.845567 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:51.845537 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:52.161468 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.161431 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9"] Oct 30 21:53:52.164862 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:53:52.164765 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86590685_ff39_41c3_a263_b54d75714605.slice/crio-54dd9cd9ee7b287ef3fcf12f4773dab04c69b69e9df9ea807d309ba5274da9f2 WatchSource:0}: Error finding container 54dd9cd9ee7b287ef3fcf12f4773dab04c69b69e9df9ea807d309ba5274da9f2: Status 404 returned error can't find the container with id 54dd9cd9ee7b287ef3fcf12f4773dab04c69b69e9df9ea807d309ba5274da9f2 Oct 30 21:53:52.167083 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.167052 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:52.167170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.167113 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:52.167170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.167142 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:53:52.688020 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.687988 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:52.692835 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.692811 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:52.705165 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.705135 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" event={"ID":"86590685-ff39-41c3-a263-b54d75714605","Type":"ContainerStarted","Data":"112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01"} Oct 30 21:53:52.705275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.705181 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" event={"ID":"86590685-ff39-41c3-a263-b54d75714605","Type":"ContainerStarted","Data":"54dd9cd9ee7b287ef3fcf12f4773dab04c69b69e9df9ea807d309ba5274da9f2"} Oct 30 21:53:52.705442 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.705423 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:52.706173 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.706158 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:53:52.724947 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.724925 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:52.930945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:52.930893 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podStartSLOduration=1.9308793419999999 podStartE2EDuration="1.930879342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:52.929413808 +0000 UTC m=+643.118951505" watchObservedRunningTime="2026-10-30 21:53:52.930879342 +0000 UTC m=+643.120417017" Oct 30 21:53:53.718146 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:53.718117 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:53.730209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:53.730183 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:53.897017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:53.896984 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:54.078998 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:54.078969 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:54.405005 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:54.404930 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:54.714454 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:54.714366 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:54.715206 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:54.715188 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:53:54.810328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:54.810266 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.085950 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:55.085889 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.298091 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:55.298053 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.490863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:55.490784 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.996448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:55.996410 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:56.209138 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:56.209106 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:58.501828 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:58.501789 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:58.685353 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:53:58.685272 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:00.242516 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:00.242479 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:00.706614 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:00.706575 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:00.930611 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:00.930575 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.802002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:02.801939 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.989867 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:02.989829 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.169771 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.169728 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.422382 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.422271 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.675719 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.675593 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.868201 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.868165 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.885360 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.885330 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd"] Oct 30 21:54:03.889017 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.889001 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:03.941183 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.941114 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b0c55f-dockercfg-fgkfx\"" Oct 30 21:54:03.941183 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:03.941166 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b0c55f\"" Oct 30 21:54:04.001843 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.001793 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd"] Oct 30 21:54:04.010033 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.009997 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8"] Oct 30 21:54:04.014401 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.014366 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.033558 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.033531 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.033689 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.033583 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59mjq\" (UniqueName: \"kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq\") pod \"payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.073413 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.073384 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8"] Oct 30 21:54:04.134585 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.134556 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.134741 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.134601 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x28l\" (UniqueName: \"kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l\") pod \"payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.134741 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.134634 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59mjq\" (UniqueName: \"kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq\") pod \"payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.134741 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.134664 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.135335 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.135290 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.191177 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.191149 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59mjq\" (UniqueName: \"kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq\") pod \"payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.201917 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.201857 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:04.235841 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.235809 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.235999 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.235956 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x28l\" (UniqueName: \"kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l\") pod \"payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.236706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.236683 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.293170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.293136 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x28l\" (UniqueName: \"kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l\") pod \"payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.327504 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.327479 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:04.597252 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.597225 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd"] Oct 30 21:54:04.598886 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:04.598856 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d41ced4_4605_46eb_a5bd_33c6ac8b3ecc.slice/crio-cf758bcc0f9ec03817fbcb1c2f98a44acb05680868a2fadb15aeb0b2e6079452 WatchSource:0}: Error finding container cf758bcc0f9ec03817fbcb1c2f98a44acb05680868a2fadb15aeb0b2e6079452: Status 404 returned error can't find the container with id cf758bcc0f9ec03817fbcb1c2f98a44acb05680868a2fadb15aeb0b2e6079452 Oct 30 21:54:04.600676 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.600662 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:54:04.701965 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.701942 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8"] Oct 30 21:54:04.704410 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:04.704388 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56303faa_1a27_4e35_992e_99c463aa4829.slice/crio-9b20c95c4a7fb916ce1c87bfe0d17cce2d1b4a750bdaea87af8c77c0124bc323 WatchSource:0}: Error finding container 9b20c95c4a7fb916ce1c87bfe0d17cce2d1b4a750bdaea87af8c77c0124bc323: Status 404 returned error can't find the container with id 9b20c95c4a7fb916ce1c87bfe0d17cce2d1b4a750bdaea87af8c77c0124bc323 Oct 30 21:54:04.765592 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.765562 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" event={"ID":"56303faa-1a27-4e35-992e-99c463aa4829","Type":"ContainerStarted","Data":"9b20c95c4a7fb916ce1c87bfe0d17cce2d1b4a750bdaea87af8c77c0124bc323"} Oct 30 21:54:04.766711 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:04.766688 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" event={"ID":"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc","Type":"ContainerStarted","Data":"cf758bcc0f9ec03817fbcb1c2f98a44acb05680868a2fadb15aeb0b2e6079452"} Oct 30 21:54:05.608202 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.608167 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v"] Oct 30 21:54:05.611931 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.611908 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.651196 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.651169 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.651344 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.651208 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r987v\" (UniqueName: \"kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.651407 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.651361 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.651537 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.651509 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.663681 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.663663 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:54:05.713084 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.713051 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:54:05.713396 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.713356 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-9f8aea-serving-cert\"" Oct 30 21:54:05.713516 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.713412 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:54:05.713578 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.713516 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjcd6\"" Oct 30 21:54:05.715644 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.715623 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v"] Oct 30 21:54:05.760789 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.760760 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.760923 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.760865 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.760923 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.760891 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.760923 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.760911 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r987v\" (UniqueName: \"kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.761096 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:05.761065 5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-serving-cert: secret "maas-api-e2e-worker-w5-9f8aea-serving-cert" not found Oct 30 21:54:05.761186 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:05.761173 5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls podName:03672869-8c66-4958-8597-8be4a451981b nodeName:}" failed. No retries permitted until 2026-10-30 21:54:06.261154689 +0000 UTC m=+656.450692342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls") pod "maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" (UID: "03672869-8c66-4958-8597-8be4a451981b") : secret "maas-api-e2e-worker-w5-9f8aea-serving-cert" not found Oct 30 21:54:05.761574 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.761553 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.765081 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.765058 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.822263 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.822229 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r987v\" (UniqueName: \"kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:05.839116 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:05.839080 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.018758 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:06.018664 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.189122 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:06.189081 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.266200 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:06.266161 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:06.287828 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:06.287729 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:06.528847 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:06.528815 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:06.934518 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:06.934485 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v"] Oct 30 21:54:06.937863 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:06.937822 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03672869_8c66_4958_8597_8be4a451981b.slice/crio-e579698b6a9f84ba26a2923be4663ce16df63c3b4b7cb7adb9bc6c8d35672b74 WatchSource:0}: Error finding container e579698b6a9f84ba26a2923be4663ce16df63c3b4b7cb7adb9bc6c8d35672b74: Status 404 returned error can't find the container with id e579698b6a9f84ba26a2923be4663ce16df63c3b4b7cb7adb9bc6c8d35672b74 Oct 30 21:54:07.801247 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:07.801137 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" event={"ID":"56303faa-1a27-4e35-992e-99c463aa4829","Type":"ContainerStarted","Data":"13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23"} Oct 30 21:54:07.801247 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:07.801228 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:07.802716 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:07.802687 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" event={"ID":"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc","Type":"ContainerStarted","Data":"add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2"} Oct 30 21:54:07.802852 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:07.802747 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:07.803879 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:07.803859 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" event={"ID":"03672869-8c66-4958-8597-8be4a451981b","Type":"ContainerStarted","Data":"e579698b6a9f84ba26a2923be4663ce16df63c3b4b7cb7adb9bc6c8d35672b74"} Oct 30 21:54:07.917176 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:07.917121 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" podStartSLOduration=2.157316293 podStartE2EDuration="4.91710762s" totalImagesPullingTime="2.759791327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:07.915558187 +0000 UTC m=+658.105095882" watchObservedRunningTime="2026-10-30 21:54:07.91710762 +0000 UTC m=+658.106645324" Oct 30 21:54:08.038026 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:08.037964 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" podStartSLOduration=2.180837212 podStartE2EDuration="5.03794457s" totalImagesPullingTime="2.857107358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:08.036833092 +0000 UTC m=+658.226370787" watchObservedRunningTime="2026-10-30 21:54:08.03794457 +0000 UTC m=+658.227482247" Oct 30 21:54:08.537959 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:08.537898 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:08.730712 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:08.730677 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:09.085213 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.085188 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf"] Oct 30 21:54:09.090177 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.090160 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.145547 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.145519 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf"] Oct 30 21:54:09.192772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.192741 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d516c4\"" Oct 30 21:54:09.192772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.192741 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d516c4-dockercfg-fzchv\"" Oct 30 21:54:09.204169 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.204144 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2sp\" (UniqueName: \"kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp\") pod \"payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.204437 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.204408 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.305707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.305626 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2sp\" (UniqueName: \"kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp\") pod \"payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.305836 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.305710 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.306338 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.306300 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.362281 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.362254 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2sp\" (UniqueName: \"kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp\") pod \"payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.403064 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.403032 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:09.816420 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.816369 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" event={"ID":"03672869-8c66-4958-8597-8be4a451981b","Type":"ContainerStarted","Data":"366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813"} Oct 30 21:54:09.816673 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.816645 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:09.817676 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.817658 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf"] Oct 30 21:54:09.820824 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:09.820793 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f6e1fe_0ef9_4e24_a265_ea35721222da.slice/crio-57affe14c12223465357a28b05c488d9ef7f6b85e23b68f1c49b4691a9766c43 WatchSource:0}: Error finding container 57affe14c12223465357a28b05c488d9ef7f6b85e23b68f1c49b4691a9766c43: Status 404 returned error can't find the container with id 57affe14c12223465357a28b05c488d9ef7f6b85e23b68f1c49b4691a9766c43 Oct 30 21:54:09.934809 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:09.934750 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" podStartSLOduration=2.82935862 podStartE2EDuration="4.934735534s" totalImagesPullingTime="2.105376914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.932669592 +0000 UTC m=+660.122207280" watchObservedRunningTime="2026-10-30 21:54:09.934735534 +0000 UTC m=+660.124273208" Oct 30 21:54:10.371749 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:10.371715 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:54:10.822915 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:10.822879 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" event={"ID":"f1f6e1fe-0ef9-4e24-a265-ea35721222da","Type":"ContainerStarted","Data":"7beab5db64d3e506929f25cffe6a97180312fcbdd37dc0d2d6ff943bd7788395"} Oct 30 21:54:10.823097 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:10.822946 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" event={"ID":"f1f6e1fe-0ef9-4e24-a265-ea35721222da","Type":"ContainerStarted","Data":"57affe14c12223465357a28b05c488d9ef7f6b85e23b68f1c49b4691a9766c43"} Oct 30 21:54:10.823167 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:10.823136 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:10.958446 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:10.958385 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" podStartSLOduration=1.9583702060000001 podStartE2EDuration="1.958370206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.957099705 +0000 UTC m=+661.146637413" watchObservedRunningTime="2026-10-30 21:54:10.958370206 +0000 UTC m=+661.147907899" Oct 30 21:54:11.030866 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.030827 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.215467 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.215377 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.398329 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.396635 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.588686 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.585577 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt"] Oct 30 21:54:11.600389 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.594340 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.605608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.605572 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.631874 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.631854 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tskh\" (UniqueName: \"kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh\") pod \"payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.631982 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.631889 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.643150 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.643121 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt"] Oct 30 21:54:11.658252 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.658226 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-29cbad-dockercfg-449h4\"" Oct 30 21:54:11.671914 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.671890 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-29cbad\"" Oct 30 21:54:11.732375 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.732339 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tskh\" (UniqueName: \"kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh\") pod \"payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.732542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.732387 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.733069 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.733049 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.802412 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.802371 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tskh\" (UniqueName: \"kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh\") pod \"payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:11.923126 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:11.923041 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:12.324891 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:12.324862 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt"] Oct 30 21:54:12.328053 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:12.328022 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a354e8_e5ca_4894_9695_e6e14d1b9cfb.slice/crio-fca9c82563c2570914396862e703b55a9ab4747b56d81f249546a97b77b94316 WatchSource:0}: Error finding container fca9c82563c2570914396862e703b55a9ab4747b56d81f249546a97b77b94316: Status 404 returned error can't find the container with id fca9c82563c2570914396862e703b55a9ab4747b56d81f249546a97b77b94316 Oct 30 21:54:12.836531 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:12.836489 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" event={"ID":"02a354e8-e5ca-4894-9695-e6e14d1b9cfb","Type":"ContainerStarted","Data":"1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4"} Oct 30 21:54:12.837002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:12.836555 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:12.837002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:12.836573 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" event={"ID":"02a354e8-e5ca-4894-9695-e6e14d1b9cfb","Type":"ContainerStarted","Data":"fca9c82563c2570914396862e703b55a9ab4747b56d81f249546a97b77b94316"} Oct 30 21:54:12.958002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:12.957949 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" podStartSLOduration=1.957935781 podStartE2EDuration="1.957935781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.957086418 +0000 UTC m=+663.146624086" watchObservedRunningTime="2026-10-30 21:54:12.957935781 +0000 UTC m=+663.147473458" Oct 30 21:54:15.829903 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:15.829869 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 21:54:15.908068 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:15.908039 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:16.141300 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.141228 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:16.323872 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.323832 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt"] Oct 30 21:54:16.328146 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.328111 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.380671 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.380635 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.380838 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.380757 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445vt\" (UniqueName: \"kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt\") pod \"payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.381328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.381296 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-0f42cb-dockercfg-xp276\"" Oct 30 21:54:16.381392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.381337 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-0f42cb\"" Oct 30 21:54:16.390752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.390723 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt"] Oct 30 21:54:16.442776 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.442696 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694"] Oct 30 21:54:16.446381 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.446365 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.482479 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.482439 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.482632 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.482549 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.482632 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.482605 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.482718 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.482689 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445vt\" (UniqueName: \"kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt\") pod \"payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.482753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.482712 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.482849 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.482815 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ndzg\" (UniqueName: \"kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.483255 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.483236 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.498174 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.498140 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-29cbad-serving-cert\"" Oct 30 21:54:16.509854 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.509823 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694"] Oct 30 21:54:16.540395 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.540366 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445vt\" (UniqueName: \"kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt\") pod \"payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.584315 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.584280 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.584489 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.584351 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.584489 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.584380 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.584489 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.584456 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ndzg\" (UniqueName: \"kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.584937 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:16.584737 5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-serving-cert: secret "maas-api-e2e-worker-w6-29cbad-serving-cert" not found Oct 30 21:54:16.584937 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:16.584796 5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls podName:57403d72-cdb9-4503-ac30-c0c5d341ee49 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:17.084780397 +0000 UTC m=+667.274318050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls") pod "maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" (UID: "57403d72-cdb9-4503-ac30-c0c5d341ee49") : secret "maas-api-e2e-worker-w6-29cbad-serving-cert" not found Oct 30 21:54:16.585569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.585506 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.587608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.587561 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.641791 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.641759 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ndzg\" (UniqueName: \"kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:16.641962 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.641940 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:16.674048 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.674018 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:16.851822 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:16.851788 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.023005 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.022981 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt"] Oct 30 21:54:17.024684 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:17.024653 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f665cb_704b_49b8_833f_3015ad15c1cb.slice/crio-3a162dfe5ac3c9e9eb7d6142f934df85eb0418f280c6a26eda72e123ab235f4d WatchSource:0}: Error finding container 3a162dfe5ac3c9e9eb7d6142f934df85eb0418f280c6a26eda72e123ab235f4d: Status 404 returned error can't find the container with id 3a162dfe5ac3c9e9eb7d6142f934df85eb0418f280c6a26eda72e123ab235f4d Oct 30 21:54:17.090506 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.090477 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:17.092857 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.092831 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\") pod \"maas-api-e2e-worker-w6-29cbad-b6587449d-v8694\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:17.360654 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.360608 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:17.735706 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.735620 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694"] Oct 30 21:54:17.740795 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:17.740753 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57403d72_cdb9_4503_ac30_c0c5d341ee49.slice/crio-80e5096750a3f2c60ab9eca3bc12cce8cc3f6bf496707cd1edead5ce03bf7e4c WatchSource:0}: Error finding container 80e5096750a3f2c60ab9eca3bc12cce8cc3f6bf496707cd1edead5ce03bf7e4c: Status 404 returned error can't find the container with id 80e5096750a3f2c60ab9eca3bc12cce8cc3f6bf496707cd1edead5ce03bf7e4c Oct 30 21:54:17.866181 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.866143 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" event={"ID":"57403d72-cdb9-4503-ac30-c0c5d341ee49","Type":"ContainerStarted","Data":"80e5096750a3f2c60ab9eca3bc12cce8cc3f6bf496707cd1edead5ce03bf7e4c"} Oct 30 21:54:17.867867 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.867839 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" event={"ID":"d9f665cb-704b-49b8-833f-3015ad15c1cb","Type":"ContainerStarted","Data":"68ef0f17df7fcc7517f16ce75fcdd0acfa27a4265cd79fccbf038bf10729d2fd"} Oct 30 21:54:17.867989 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.867895 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" event={"ID":"d9f665cb-704b-49b8-833f-3015ad15c1cb","Type":"ContainerStarted","Data":"3a162dfe5ac3c9e9eb7d6142f934df85eb0418f280c6a26eda72e123ab235f4d"} Oct 30 21:54:17.868072 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.868058 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:17.986676 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:17.986558 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" podStartSLOduration=1.986545012 podStartE2EDuration="1.986545012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:17.984284732 +0000 UTC m=+668.173822406" watchObservedRunningTime="2026-10-30 21:54:17.986545012 +0000 UTC m=+668.176082686" Oct 30 21:54:18.481012 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.480940 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:18.661104 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.661071 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:18.811725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.811655 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 21:54:18.811725 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.811717 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 21:54:18.874095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.874062 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" event={"ID":"57403d72-cdb9-4503-ac30-c0c5d341ee49","Type":"ContainerStarted","Data":"6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c"} Oct 30 21:54:18.874572 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.874120 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:18.875701 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.875680 5508 generic.go:356] "Generic (PLEG): container finished" podID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerID="559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85" exitCode=0 Oct 30 21:54:18.875808 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.875713 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" event={"ID":"d73eb82d-1732-4b29-b53a-104b03d75e32","Type":"ContainerDied","Data":"559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85"} Oct 30 21:54:18.876092 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.876077 5508 scope.go:122] "RemoveContainer" containerID="559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85" Oct 30 21:54:18.877183 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.877154 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:18.877279 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.877231 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:18.877362 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:18.877280 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:19.284333 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:19.283360 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" podStartSLOduration=2.488362496 podStartE2EDuration="3.283337596s" totalImagesPullingTime="794.9751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:19.281470106 +0000 UTC m=+669.471007798" watchObservedRunningTime="2026-10-30 21:54:19.283337596 +0000 UTC m=+669.472875266" Oct 30 21:54:19.905899 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:19.905866 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" event={"ID":"d73eb82d-1732-4b29-b53a-104b03d75e32","Type":"ContainerStarted","Data":"a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f"} Oct 30 21:54:20.040481 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:20.040450 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:54:20.041832 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:20.041804 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:54:20.041962 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:20.041852 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 21:54:21.039978 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:21.039944 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:54:21.040382 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:21.039990 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 21:54:21.699443 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:21.699400 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:54:21.831810 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:21.831786 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 21:54:22.040258 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:22.040178 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:54:22.040258 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:22.040229 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 21:54:22.300260 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:22.300161 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:23.040865 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:23.040826 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:54:23.041257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:23.040892 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 21:54:23.845618 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:23.845591 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 21:54:24.041006 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:24.040970 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 21:54:24.041514 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:24.041020 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 21:54:24.526053 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:24.526015 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:24.911816 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:24.911790 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 21:54:25.101638 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:25.101602 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:54:25.104635 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:25.104604 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 21:54:27.554605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.554572 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw"] Oct 30 21:54:27.558052 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.558035 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.609600 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.609571 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-0c99b1-serving-cert\"" Oct 30 21:54:27.636215 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.636186 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.636379 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.636248 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2znx\" (UniqueName: \"kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.636423 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.636380 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.636474 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.636463 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.660600 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.660567 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw"] Oct 30 21:54:27.737042 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.737013 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.737230 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.737074 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2znx\" (UniqueName: \"kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.737230 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.737099 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.737230 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.737144 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.737438 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:27.737244 5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-serving-cert: secret "maas-api-e2e-worker-w2-0c99b1-serving-cert" not found Oct 30 21:54:27.737438 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:27.737326 5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls podName:2ddf0438-70cf-448d-9510-4fe6380d9937 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:28.2372883 +0000 UTC m=+678.426825967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls") pod "maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" (UID: "2ddf0438-70cf-448d-9510-4fe6380d9937") : secret "maas-api-e2e-worker-w2-0c99b1-serving-cert" not found Oct 30 21:54:27.737704 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.737683 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.739720 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.739698 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.800396 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.800369 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2znx\" (UniqueName: \"kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:27.897646 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:27.897588 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:28.058116 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.058087 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:28.218561 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.218486 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:28.242093 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.242059 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:28.244791 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.244767 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\") pod \"maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:28.469936 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.469866 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:28.848753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.848715 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw"] Oct 30 21:54:28.852443 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:28.852416 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddf0438_70cf_448d_9510_4fe6380d9937.slice/crio-1b1e4f5dc69f7c3de6ca576b72a487dcd2992261cc3179c76217d968f600fca9 WatchSource:0}: Error finding container 1b1e4f5dc69f7c3de6ca576b72a487dcd2992261cc3179c76217d968f600fca9: Status 404 returned error can't find the container with id 1b1e4f5dc69f7c3de6ca576b72a487dcd2992261cc3179c76217d968f600fca9 Oct 30 21:54:28.879613 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.879588 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 21:54:28.957086 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:28.957044 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" event={"ID":"2ddf0438-70cf-448d-9510-4fe6380d9937","Type":"ContainerStarted","Data":"1b1e4f5dc69f7c3de6ca576b72a487dcd2992261cc3179c76217d968f600fca9"} Oct 30 21:54:29.963472 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:29.963379 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" event={"ID":"2ddf0438-70cf-448d-9510-4fe6380d9937","Type":"ContainerStarted","Data":"a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc"} Oct 30 21:54:29.963472 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:29.963454 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:30.091225 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:30.091170 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" podStartSLOduration=2.259137568 podStartE2EDuration="3.091154386s" totalImagesPullingTime="832.016818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:30.089518422 +0000 UTC m=+680.279056114" watchObservedRunningTime="2026-10-30 21:54:30.091154386 +0000 UTC m=+680.280692060" Oct 30 21:54:31.974431 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:31.974398 5508 generic.go:356] "Generic (PLEG): container finished" podID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerID="8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807" exitCode=0 Oct 30 21:54:31.974864 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:31.974463 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" event={"ID":"0c99a268-d806-4b8f-8aee-824e3a6d6b1a","Type":"ContainerDied","Data":"8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807"} Oct 30 21:54:31.974864 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:31.974769 5508 scope.go:122] "RemoveContainer" containerID="8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807" Oct 30 21:54:31.976103 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:31.976076 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:31.976162 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:31.976146 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:31.976196 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:31.976177 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:54:33.006470 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:33.006429 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" event={"ID":"0c99a268-d806-4b8f-8aee-824e3a6d6b1a","Type":"ContainerStarted","Data":"6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3"} Oct 30 21:54:33.066278 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:33.066238 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:33.695867 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:33.695839 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:54:33.696193 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:33.696173 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:54:33.696358 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:33.696334 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:54:34.699334 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:34.694703 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:54:34.699334 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:34.694768 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:54:35.691328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:35.689709 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:54:35.691328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:35.689768 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:54:35.974056 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:35.973981 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 21:54:36.695746 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:36.695708 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:54:36.696063 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:36.696023 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:54:37.695329 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.694679 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:54:37.695329 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.694742 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:54:37.706292 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.706263 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:54:37.732350 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.732225 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb"] Oct 30 21:54:37.737201 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.737177 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.746546 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.746525 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.746647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.746560 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8xk\" (UniqueName: \"kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.746733 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.746714 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.746838 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.746818 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.797277 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.797242 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-0f42cb-serving-cert\"" Oct 30 21:54:37.803278 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.803244 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb"] Oct 30 21:54:37.847425 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.847395 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.847582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.847439 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.847582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.847498 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.847582 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.847520 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8xk\" (UniqueName: \"kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.847582 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:37.847550 5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-serving-cert: secret "maas-api-e2e-worker-w3-0f42cb-serving-cert" not found Oct 30 21:54:37.847775 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:37.847620 5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls podName:a65189ce-f5c3-4c0b-bf1d-cebae139b17b nodeName:}" failed. No retries permitted until 2026-10-30 21:54:38.347603817 +0000 UTC m=+688.537141491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls") pod "maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" (UID: "a65189ce-f5c3-4c0b-bf1d-cebae139b17b") : secret "maas-api-e2e-worker-w3-0f42cb-serving-cert" not found Oct 30 21:54:37.848231 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.848208 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.850255 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.850216 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:37.907489 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:37.907450 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8xk\" (UniqueName: \"kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:38.010257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.010155 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:38.335192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.335160 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:38.351238 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.351209 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:38.353867 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.353847 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:38.648765 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.648684 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:38.692525 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.692492 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:54:38.693450 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:38.693431 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 21:54:39.071477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:39.071454 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb"] Oct 30 21:54:39.073803 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:54:39.073776 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65189ce_f5c3_4c0b_bf1d_cebae139b17b.slice/crio-5f944dad8d4fff88c45403446f8ab173d924e70c2ceba969597f351244cd07ec WatchSource:0}: Error finding container 5f944dad8d4fff88c45403446f8ab173d924e70c2ceba969597f351244cd07ec: Status 404 returned error can't find the container with id 5f944dad8d4fff88c45403446f8ab173d924e70c2ceba969597f351244cd07ec Oct 30 21:54:39.464495 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:39.463642 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.045124 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:40.045090 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" event={"ID":"a65189ce-f5c3-4c0b-bf1d-cebae139b17b","Type":"ContainerStarted","Data":"f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc"} Oct 30 21:54:40.045327 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:40.045145 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" event={"ID":"a65189ce-f5c3-4c0b-bf1d-cebae139b17b","Type":"ContainerStarted","Data":"5f944dad8d4fff88c45403446f8ab173d924e70c2ceba969597f351244cd07ec"} Oct 30 21:54:40.045327 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:40.045183 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:40.164831 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:40.164736 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" podStartSLOduration=2.383296612 podStartE2EDuration="3.164721298s" totalImagesPullingTime="781.424686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:40.162356517 +0000 UTC m=+690.351894192" watchObservedRunningTime="2026-10-30 21:54:40.164721298 +0000 UTC m=+690.354258972" Oct 30 21:54:40.372730 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:54:40.372694 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:54:41.129908 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:41.129882 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:41.327199 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:41.327171 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:41.518542 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:41.518462 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:43.540156 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:43.540088 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:43.713350 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:43.713288 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:43.870466 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:43.870421 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:45.096738 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:45.096705 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:46.056347 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:46.056299 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 21:54:49.133892 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:49.133860 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.780060 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:54.780022 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:55.645081 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:55.645046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.398808 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:57.398741 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.610358 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:57.610325 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.815562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:54:57.815476 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:00.113159 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:00.113123 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:00.395383 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:00.395278 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:02.500212 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:02.500164 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:03.132355 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:03.132283 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:03.300121 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:03.300082 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:05.433412 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:05.433373 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:06.881215 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:06.881182 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:07.463045 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:07.463007 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:07.630727 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:07.630689 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:07.815372 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:07.815264 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:08.190427 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.190387 5508 generic.go:356] "Generic (PLEG): container finished" podID="86590685-ff39-41c3-a263-b54d75714605" containerID="112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01" exitCode=0 Oct 30 21:55:08.190852 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.190457 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" event={"ID":"86590685-ff39-41c3-a263-b54d75714605","Type":"ContainerDied","Data":"112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01"} Oct 30 21:55:08.190899 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.190851 5508 scope.go:122] "RemoveContainer" containerID="112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01" Oct 30 21:55:08.192225 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.192199 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:08.192292 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.192267 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:08.192353 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.192298 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:55:08.593690 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:08.593639 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:09.206199 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:09.206163 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" event={"ID":"86590685-ff39-41c3-a263-b54d75714605","Type":"ContainerStarted","Data":"2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2"} Oct 30 21:55:09.715199 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:09.715155 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:55:09.724362 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:09.724343 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:55:09.725491 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:09.725472 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:55:09.725538 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:09.725516 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:55:10.372729 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:55:10.372695 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:55:10.725234 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:10.725150 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:55:10.725234 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:10.725200 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:55:10.752210 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:10.752179 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.724493 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:11.724444 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:55:11.724989 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:11.724507 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:55:12.725129 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:12.725095 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:55:12.725535 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:12.725160 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:55:13.724793 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:13.724757 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:55:13.724966 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:13.724811 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:55:14.728694 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:14.728666 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:55:14.729605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:14.729586 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 21:55:15.975972 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:15.975934 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:20.823945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:20.823907 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:20.991101 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:20.991064 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.630971 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:21.630901 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.827821 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:21.827789 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:24.864655 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:24.864624 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:25.870448 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:25.870415 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:26.293647 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:26.293517 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:29.427267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:29.427231 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:30.046608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:30.046566 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:30.212719 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:30.212684 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:31.096715 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:31.096680 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:33.874946 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:33.874914 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:34.466454 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:34.466417 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:34.641940 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:34.641904 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:36.513853 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:36.513792 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:38.159488 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:38.159441 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:39.631085 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:39.631021 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:40.375507 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:55:40.375469 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:55:41.315242 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:41.315212 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:41.516022 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:41.515982 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:46.405566 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:46.405527 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:46.694989 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:46.694889 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:51.387428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:51.387398 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.186948 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:55:56.186915 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:01.264753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:01.264709 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:06.019540 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:06.019507 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:06.502462 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:06.502431 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:10.372055 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:56:10.372013 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:56:11.115995 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:11.115963 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:15.972549 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:15.972515 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:16.334153 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:16.334120 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:20.780078 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:20.780046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:25.215184 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:25.215147 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:25.395847 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:25.395814 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:25.662340 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:25.662273 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:27.697657 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.697626 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt"] Oct 30 21:56:27.701283 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.701266 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.809277 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.809239 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"premium-w5-9f8aea-kserve-self-signed-certs\"" Oct 30 21:56:27.809277 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.809247 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"default-dockercfg-vm2j7\"" Oct 30 21:56:27.809539 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.809409 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"openshift-service-ca.crt\"" Oct 30 21:56:27.809539 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.809416 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"kube-root-ca.crt\"" Oct 30 21:56:27.811072 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.811054 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt"] Oct 30 21:56:27.820232 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.820213 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.820295 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.820277 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz688\" (UniqueName: \"kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.820403 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.820384 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.820438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.820418 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.820518 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.820504 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.820556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.820535 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.859568 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.859525 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:27.922036 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922007 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz688\" (UniqueName: \"kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922047 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922075 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922109 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922125 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922176 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922508 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922671 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922544 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.922671 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.922657 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.924479 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.924462 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.924746 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.924729 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:27.988273 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:27.988215 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz688\" (UniqueName: \"kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688\") pod \"premium-w5-9f8aea-kserve-86647694d7-7jhlt\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:28.012766 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:28.012743 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:28.402332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:28.402294 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt"] Oct 30 21:56:28.404474 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:56:28.404445 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34d1011f_485e_4152_af59_fc2f37d02649.slice/crio-c75285717c1370fa15dfff88bd5e97ea03b0628b845819efd2e94978675fc812 WatchSource:0}: Error finding container c75285717c1370fa15dfff88bd5e97ea03b0628b845819efd2e94978675fc812: Status 404 returned error can't find the container with id c75285717c1370fa15dfff88bd5e97ea03b0628b845819efd2e94978675fc812 Oct 30 21:56:28.622110 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:28.622078 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" event={"ID":"34d1011f-485e-4152-af59-fc2f37d02649","Type":"ContainerStarted","Data":"88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767"} Oct 30 21:56:28.622276 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:28.622122 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" event={"ID":"34d1011f-485e-4152-af59-fc2f37d02649","Type":"ContainerStarted","Data":"c75285717c1370fa15dfff88bd5e97ea03b0628b845819efd2e94978675fc812"} Oct 30 21:56:28.747608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:28.747509 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podStartSLOduration=1.747494503 podStartE2EDuration="1.747494503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:28.745903744 +0000 UTC m=+798.935441434" watchObservedRunningTime="2026-10-30 21:56:28.747494503 +0000 UTC m=+798.937032177" Oct 30 21:56:29.013722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:29.013653 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:30.865924 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:30.865887 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:35.679279 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:35.679241 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:38.012902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:38.012868 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:38.026238 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:38.026213 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:38.683277 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:38.683247 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 21:56:39.856717 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.856684 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp"] Oct 30 21:56:39.860709 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.860690 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.914439 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.914413 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"kube-root-ca.crt\"" Oct 30 21:56:39.939935 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.939902 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.940089 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.939941 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.940089 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.940060 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.940187 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.940109 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.940238 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.940217 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.940288 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.940275 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl24h\" (UniqueName: \"kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:39.963721 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.963697 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"openshift-service-ca.crt\"" Oct 30 21:56:39.963863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.963763 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"premium-w1-d516c4-kserve-self-signed-certs\"" Oct 30 21:56:39.963863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.963765 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"default-dockercfg-t65jc\"" Oct 30 21:56:39.965681 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.965663 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp"] Oct 30 21:56:39.969967 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:39.969947 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:40.041223 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041190 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041223 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041224 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041511 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041266 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041511 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041290 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041511 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041344 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041511 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041389 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl24h\" (UniqueName: \"kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041724 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041634 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041724 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041699 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.041835 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.041768 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.043915 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.043898 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.044562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.044542 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.097780 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.097749 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl24h\" (UniqueName: \"kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h\") pod \"premium-w1-d516c4-kserve-85df847db4-f7pkp\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.173779 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.173696 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:40.372220 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:56:40.372187 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:56:40.548254 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.548224 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp"] Oct 30 21:56:40.550367 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:56:40.550341 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc6e41a3_3330_4d4d_afb0_7c175c53f4d4.slice/crio-a7cb12ba1d85fb80dc09f04e4500b716a94663f57d176234d4ff27641250e9f6 WatchSource:0}: Error finding container a7cb12ba1d85fb80dc09f04e4500b716a94663f57d176234d4ff27641250e9f6: Status 404 returned error can't find the container with id a7cb12ba1d85fb80dc09f04e4500b716a94663f57d176234d4ff27641250e9f6 Oct 30 21:56:40.649235 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.649198 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:40.685328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.685268 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" event={"ID":"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4","Type":"ContainerStarted","Data":"1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a"} Oct 30 21:56:40.685328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.685330 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" event={"ID":"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4","Type":"ContainerStarted","Data":"a7cb12ba1d85fb80dc09f04e4500b716a94663f57d176234d4ff27641250e9f6"} Oct 30 21:56:40.800910 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:40.800819 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podStartSLOduration=1.8008048620000001 podStartE2EDuration="1.800804862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:40.799049505 +0000 UTC m=+810.988587197" watchObservedRunningTime="2026-10-30 21:56:40.800804862 +0000 UTC m=+810.990342563" Oct 30 21:56:41.174690 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:41.174653 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:45.467357 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:45.467323 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.174497 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:50.174412 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:50.188471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:50.188447 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:50.266903 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:50.266873 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.747079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:50.747050 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 21:56:55.126659 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:55.126606 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.912653 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:56:59.912621 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:04.760861 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:04.760826 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:08.966734 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:08.966698 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.768960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:09.768927 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:10.372910 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:57:10.372872 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:57:14.600796 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:14.600759 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:19.411873 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:19.411831 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:20.603440 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:20.603406 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:24.369666 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:24.369629 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.554939 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:25.554908 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:29.172574 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:29.172540 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:29.374974 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:29.374939 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.167782 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.167742 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.718079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.718045 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m"] Oct 30 21:57:34.721786 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.721770 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.773613 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.773588 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"openshift-service-ca.crt\"" Oct 30 21:57:34.822979 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.822934 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"kube-root-ca.crt\"" Oct 30 21:57:34.823159 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.823013 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"premium-w0-dde1e8-kserve-self-signed-certs\"" Oct 30 21:57:34.823159 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.823072 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"default-dockercfg-zthlq\"" Oct 30 21:57:34.824890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.824868 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m"] Oct 30 21:57:34.826477 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.826458 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.826522 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.826496 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.826559 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.826522 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69695\" (UniqueName: \"kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.826664 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.826648 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.826711 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.826699 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.826779 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.826767 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.832875 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.832854 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.928361 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928298 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928394 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928445 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928478 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928500 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69695\" (UniqueName: \"kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928544 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928542 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928802 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928721 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928854 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928833 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.928920 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.928903 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.930831 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.930805 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.931091 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.931075 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:34.988584 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:34.988528 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69695\" (UniqueName: \"kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695\") pod \"premium-w0-dde1e8-kserve-546445bc9-2dq6m\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:35.033948 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:35.033921 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:35.414890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:35.414867 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m"] Oct 30 21:57:35.417084 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:57:35.417059 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dd33b67_01de_45df_a7af_2920af1ca4ec.slice/crio-02d5c1b7fde236126b31bfa5fa5b98be0f7392ffd1d87860bdda7456bfec0b48 WatchSource:0}: Error finding container 02d5c1b7fde236126b31bfa5fa5b98be0f7392ffd1d87860bdda7456bfec0b48: Status 404 returned error can't find the container with id 02d5c1b7fde236126b31bfa5fa5b98be0f7392ffd1d87860bdda7456bfec0b48 Oct 30 21:57:35.982543 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:35.982506 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" event={"ID":"5dd33b67-01de-45df-a7af-2920af1ca4ec","Type":"ContainerStarted","Data":"0c3d9cf8bb4b0c2315ac07a151c6695be0765c671d53afafcd47138b76a136c9"} Oct 30 21:57:35.982704 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:35.982557 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" event={"ID":"5dd33b67-01de-45df-a7af-2920af1ca4ec","Type":"ContainerStarted","Data":"02d5c1b7fde236126b31bfa5fa5b98be0f7392ffd1d87860bdda7456bfec0b48"} Oct 30 21:57:36.034031 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.033995 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:36.097679 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.097622 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podStartSLOduration=2.097608627 podStartE2EDuration="2.097608627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:36.095195471 +0000 UTC m=+866.284733147" watchObservedRunningTime="2026-10-30 21:57:36.097608627 +0000 UTC m=+866.287146302" Oct 30 21:57:36.893545 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.892533 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst"] Oct 30 21:57:36.900686 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.900656 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.946983 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.946955 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947153 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947005 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947153 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947078 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5m5\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947289 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947165 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947289 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947237 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947417 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947346 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947417 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947409 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947507 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947437 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.947541 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.947521 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:36.952243 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:36.952224 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9f8aea-openshift-default-dockercfg-64lqz\"" Oct 30 21:57:37.003791 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.003763 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst"] Oct 30 21:57:37.048298 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048266 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5m5\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.048466 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048321 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.048466 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048375 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.048569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048540 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.048668 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048645 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.048877 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048836 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049100 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.048771 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049135 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049167 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049203 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049267 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049263 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049374 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049641 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049609 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.049746 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.049726 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.051121 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.051103 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.052035 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.052018 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.104797 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.104776 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.104928 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.104910 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5m5\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.214195 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.214125 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:37.645473 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.645447 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst"] Oct 30 21:57:37.647644 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:57:37.647618 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bf880a_64d5_481b_8cc5_bde91bb82995.slice/crio-65b858ebaec9140370741f6b44a0464560c53c20c40249228032e6d31ab96923 WatchSource:0}: Error finding container 65b858ebaec9140370741f6b44a0464560c53c20c40249228032e6d31ab96923: Status 404 returned error can't find the container with id 65b858ebaec9140370741f6b44a0464560c53c20c40249228032e6d31ab96923 Oct 30 21:57:37.650086 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.650050 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:37.650191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.650131 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:37.650191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.650177 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:57:37.994965 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.994883 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" event={"ID":"e5bf880a-64d5-481b-8cc5-bde91bb82995","Type":"ContainerStarted","Data":"1480aabcb1100a737e3edb8eb311fd459ef3684a77ea5425db63d036624daaad"} Oct 30 21:57:37.994965 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:37.994947 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" event={"ID":"e5bf880a-64d5-481b-8cc5-bde91bb82995","Type":"ContainerStarted","Data":"65b858ebaec9140370741f6b44a0464560c53c20c40249228032e6d31ab96923"} Oct 30 21:57:38.118331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:38.115833 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podStartSLOduration=2.115811655 podStartE2EDuration="2.115811655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:38.112919085 +0000 UTC m=+868.302456762" watchObservedRunningTime="2026-10-30 21:57:38.115811655 +0000 UTC m=+868.305349331" Oct 30 21:57:38.214637 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:38.214603 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:38.216429 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:38.216401 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:57:38.216558 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:38.216451 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 21:57:38.963112 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:38.963074 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:39.215140 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:39.215058 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:57:39.215140 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:39.215112 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 21:57:40.215413 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:40.215377 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:57:40.215893 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:40.215437 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 21:57:40.372295 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:57:40.372265 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:57:41.215004 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:41.214969 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:57:41.215171 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:41.215023 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 21:57:42.214800 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:42.214634 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:57:42.214800 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:42.214704 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 21:57:43.215394 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:43.215354 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 21:57:43.215797 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:43.215416 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 21:57:43.919777 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:43.919726 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:44.345232 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:44.345205 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:44.345703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:44.345682 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:44.346369 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:44.346352 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 21:57:45.034984 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:45.034948 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:45.048524 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:45.048499 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:45.059718 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:45.059695 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 21:57:48.868840 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:48.868805 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.631074 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:52.631039 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.786735 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:52.786701 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.663162 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:53.663107 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:58.613762 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:57:58.613729 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:02.823800 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:02.823764 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm"] Oct 30 21:58:02.824333 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:02.823993 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" containerID="cri-o://eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b" gracePeriod=30 Oct 30 21:58:03.553941 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:03.553907 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:07.819749 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.819705 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn"] Oct 30 21:58:07.823865 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.823845 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.876452 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.876431 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d516c4-openshift-default-dockercfg-mtv65\"" Oct 30 21:58:07.906139 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906115 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906170 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906257 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906242 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906282 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906355 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906469 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906399 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906469 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906422 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtpzb\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906469 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906448 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.906588 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.906554 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:07.927657 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:07.927633 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn"] Oct 30 21:58:08.007885 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.007852 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008028 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.007906 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008028 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.007998 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008105 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008058 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008105 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008083 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008168 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008104 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008168 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008138 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008238 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008195 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008288 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008248 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtpzb\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008507 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008445 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008636 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008584 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008831 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008788 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.008923 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.008904 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.009179 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.009156 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.010561 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.010527 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.010764 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.010743 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.065879 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.065849 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.065990 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.065902 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtpzb\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.091224 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.091202 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:58:08.138779 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.138756 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:08.165717 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.165690 5508 generic.go:356] "Generic (PLEG): container finished" podID="05633acf-5645-4b82-81aa-c73149763847" containerID="eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b" exitCode=0 Oct 30 21:58:08.165811 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.165741 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" Oct 30 21:58:08.165811 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.165775 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" event={"ID":"05633acf-5645-4b82-81aa-c73149763847","Type":"ContainerDied","Data":"eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b"} Oct 30 21:58:08.165913 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.165831 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm" event={"ID":"05633acf-5645-4b82-81aa-c73149763847","Type":"ContainerDied","Data":"f0da7f89278b3bfb89c21fa7a07f9a0e0d4276e1093405465c529cf032dfc6e1"} Oct 30 21:58:08.165913 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.165867 5508 scope.go:122] "RemoveContainer" containerID="eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b" Oct 30 21:58:08.174753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.174734 5508 scope.go:122] "RemoveContainer" containerID="eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b" Oct 30 21:58:08.175017 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:58:08.174999 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b\": container with ID starting with eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b not found: ID does not exist" containerID="eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b" Oct 30 21:58:08.175080 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.175026 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b"} err="failed to get container status \"eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b\": rpc error: code = NotFound desc = could not find container \"eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b\": container with ID starting with eee25f38a89861ca29ac6e6f1ea36bc89d16ba47a9e2670356da0550fbaf479b not found: ID does not exist" Oct 30 21:58:08.209794 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.209759 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.209927 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.209843 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.209996 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.209942 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.210061 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210025 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.210061 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210025 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:08.210167 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210075 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.211009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210670 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.211009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210707 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.211009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210737 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:08.211009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210762 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.211009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210794 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data\") pod \"05633acf-5645-4b82-81aa-c73149763847\" (UID: \"05633acf-5645-4b82-81aa-c73149763847\") " Oct 30 21:58:08.211009 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.210880 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:08.212438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.211270 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.212438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.211291 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05633acf-5645-4b82-81aa-c73149763847-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.212438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.211347 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.212438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.212180 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:08.212438 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.212265 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:08.215371 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.215330 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:08.215872 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.215842 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "kube-api-access-tl4fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:08.217088 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.217063 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:08.217838 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.217812 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token" pod "05633acf-5645-4b82-81aa-c73149763847" (UID: "05633acf-5645-4b82-81aa-c73149763847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:08.312710 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.312682 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.312710 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.312714 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.312906 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.312730 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05633acf-5645-4b82-81aa-c73149763847-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.312906 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.312745 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl4fj\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-kube-api-access-tl4fj\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.312906 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.312760 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05633acf-5645-4b82-81aa-c73149763847-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.312906 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.312773 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05633acf-5645-4b82-81aa-c73149763847-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:08.418283 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.418192 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:08.565113 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.565087 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn"] Oct 30 21:58:08.567471 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:58:08.567445 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cb7485_ccdd_4c73_81df_45d1511c986b.slice/crio-037b28db343dbd42d33ffccb624388a88032c747d6a92ba991e618ebcc5751ee WatchSource:0}: Error finding container 037b28db343dbd42d33ffccb624388a88032c747d6a92ba991e618ebcc5751ee: Status 404 returned error can't find the container with id 037b28db343dbd42d33ffccb624388a88032c747d6a92ba991e618ebcc5751ee Oct 30 21:58:08.569710 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.569678 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:08.569773 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.569745 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:08.569808 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.569773 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:08.585960 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.585937 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm"] Oct 30 21:58:08.589703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:08.589681 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-s8pkm"] Oct 30 21:58:09.200605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:09.200511 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" event={"ID":"89cb7485-ccdd-4c73-81df-45d1511c986b","Type":"ContainerStarted","Data":"906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb"} Oct 30 21:58:09.200605 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:09.200580 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" event={"ID":"89cb7485-ccdd-4c73-81df-45d1511c986b","Type":"ContainerStarted","Data":"037b28db343dbd42d33ffccb624388a88032c747d6a92ba991e618ebcc5751ee"} Oct 30 21:58:09.319029 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:09.318976 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podStartSLOduration=2.318960288 podStartE2EDuration="2.318960288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:09.317189011 +0000 UTC m=+899.506726714" watchObservedRunningTime="2026-10-30 21:58:09.318960288 +0000 UTC m=+899.508498033" Oct 30 21:58:09.958188 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:09.958157 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05633acf-5645-4b82-81aa-c73149763847" path="/var/lib/kubelet/pods/05633acf-5645-4b82-81aa-c73149763847/volumes" Oct 30 21:58:10.139758 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:10.139725 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:10.141131 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:10.141104 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:58:10.141247 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:10.141160 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:58:10.373248 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:58:10.373211 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:58:11.139594 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:11.139560 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:58:11.139753 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:11.139609 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:58:12.139274 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:12.139231 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:58:12.139804 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:12.139287 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:58:13.139899 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:13.139861 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:58:13.140416 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:13.139929 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:58:13.297117 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:13.297000 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:14.140111 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:14.140076 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 21:58:14.140562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:14.140140 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 21:58:15.195797 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:15.195764 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:15.196562 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:15.196540 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:15.197501 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:15.197477 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 21:58:18.074126 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:18.074074 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:23.230748 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:23.230704 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:25.941158 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:25.941120 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:28.071127 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:28.071098 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:32.880394 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:32.880356 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:37.779087 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:37.779051 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.373566 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:58:40.373531 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:58:42.554938 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:42.554908 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:47.497209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:47.497168 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:47.844916 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:47.844880 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:52.362169 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:52.362129 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:57.265969 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:58:57.265935 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:02.086137 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.086098 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:02.873996 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.873964 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l"] Oct 30 21:59:02.874513 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.874498 5508 cpu_manager.go:414] "containerMap: removing container" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" Oct 30 21:59:02.874566 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.874514 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" Oct 30 21:59:02.874622 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.874612 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="05633acf-5645-4b82-81aa-c73149763847" containerName="istio-proxy" Oct 30 21:59:02.878184 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.878170 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.910708 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.910680 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nz2\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.910833 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.910737 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.910833 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.910752 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.910905 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.910865 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.910905 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.910898 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.911001 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.910987 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.911035 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.911018 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.911068 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.911057 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:02.911103 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:02.911084 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012229 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012195 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012421 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012247 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012421 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012291 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012421 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012356 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012421 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012392 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012662 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012429 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012662 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012469 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nz2\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012685 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012723 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012772 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012758 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.012937 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012882 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.013003 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.012985 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.013099 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.013078 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.013159 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.013087 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.015045 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.015025 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.015122 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.015058 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.069707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.069673 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.069707 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.069696 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nz2\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.085382 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.085288 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l"] Oct 30 21:59:03.191294 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.191207 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:03.623752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.623725 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l"] Oct 30 21:59:03.625704 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:03.625677 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b83c8_8b9c_4039_bb65_c0c180cf2dc8.slice/crio-c3e8c5d257a549fc6a3454ed9f62ff7990ac86085b37d9db4855a3b96461a959 WatchSource:0}: Error finding container c3e8c5d257a549fc6a3454ed9f62ff7990ac86085b37d9db4855a3b96461a959: Status 404 returned error can't find the container with id c3e8c5d257a549fc6a3454ed9f62ff7990ac86085b37d9db4855a3b96461a959 Oct 30 21:59:03.627827 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.627796 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:03.627897 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.627858 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:03.627897 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:03.627888 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:04.523318 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:04.523266 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" event={"ID":"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8","Type":"ContainerStarted","Data":"1dec334063968c3c83adeb5b938cced18679bfdbe9dc9f7703701d3853c3cc56"} Oct 30 21:59:04.523716 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:04.523343 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" event={"ID":"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8","Type":"ContainerStarted","Data":"c3e8c5d257a549fc6a3454ed9f62ff7990ac86085b37d9db4855a3b96461a959"} Oct 30 21:59:04.716428 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:04.716377 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podStartSLOduration=2.716362676 podStartE2EDuration="2.716362676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:04.714740522 +0000 UTC m=+954.904278216" watchObservedRunningTime="2026-10-30 21:59:04.716362676 +0000 UTC m=+954.905900352" Oct 30 21:59:04.834992 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:04.834963 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:05.193631 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:05.193495 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:05.194256 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:05.193907 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:05.194256 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:05.193963 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:06.195250 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:06.195216 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:06.195858 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:06.195828 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:07.100646 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:07.100607 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:07.192703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:07.192669 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:07.192966 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:07.192938 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:08.194719 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.194678 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:08.195323 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.195267 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:08.647446 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.647414 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k"] Oct 30 21:59:08.651317 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.651285 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.706236 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.706208 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0f42cb-openshift-default-dockercfg-7vjjp\"" Oct 30 21:59:08.757291 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.757260 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k"] Oct 30 21:59:08.799940 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.799901 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800090 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800019 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800090 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800064 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zfp\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800172 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800151 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800210 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800187 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800386 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800358 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800475 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800442 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800533 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800485 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.800592 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.800576 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901439 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901362 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901439 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901403 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901650 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901440 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901650 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901469 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901650 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901486 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zfp\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901650 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901509 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901650 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901525 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901650 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901609 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901940 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901808 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.901940 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901835 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.902051 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.901959 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.902051 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.902014 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.902146 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.902065 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.902238 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.902219 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.904147 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.904120 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.904501 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.904477 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.957875 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.957840 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zfp\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.957996 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.957897 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:08.965884 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:08.965867 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:09.199517 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.199424 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:09.199927 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.199511 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:09.431676 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.431642 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k"] Oct 30 21:59:09.458641 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:09.458603 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f08e44_69fa_4a74_9f97_5ecaf6606b55.slice/crio-529f30c20a56c8ba4f4cb6602fccfe2b8ba09c036c4659a54cef62606468b91c WatchSource:0}: Error finding container 529f30c20a56c8ba4f4cb6602fccfe2b8ba09c036c4659a54cef62606468b91c: Status 404 returned error can't find the container with id 529f30c20a56c8ba4f4cb6602fccfe2b8ba09c036c4659a54cef62606468b91c Oct 30 21:59:09.461008 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.460974 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:09.461085 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.461045 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:09.461085 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.461073 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:09.557156 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.557120 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" event={"ID":"42f08e44-69fa-4a74-9f97-5ecaf6606b55","Type":"ContainerStarted","Data":"5a26fe38e243111723223be6c27c4c3a25c065823884f63633092eadf4957a72"} Oct 30 21:59:09.557270 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.557179 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" event={"ID":"42f08e44-69fa-4a74-9f97-5ecaf6606b55","Type":"ContainerStarted","Data":"529f30c20a56c8ba4f4cb6602fccfe2b8ba09c036c4659a54cef62606468b91c"} Oct 30 21:59:09.966766 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:09.966739 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:10.194331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:10.193164 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:10.194331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:10.193228 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:10.401331 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:59:10.399020 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:59:10.968219 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:10.968185 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:59:10.968410 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:10.968239 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:59:11.192330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:11.191765 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 21:59:11.192330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:11.191833 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 21:59:11.966341 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:11.966269 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:59:11.966821 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:11.966351 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:59:12.053661 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:12.053564 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.976331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:12.975408 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:59:12.976331 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:12.975469 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:59:13.104330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:13.103601 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:13.129332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:13.104587 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:13.129332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:13.107823 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 21:59:13.287384 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:13.274213 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podStartSLOduration=5.274191841 podStartE2EDuration="5.274191841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:09.681239533 +0000 UTC m=+959.870777241" watchObservedRunningTime="2026-10-30 21:59:13.274191841 +0000 UTC m=+963.463729517" Oct 30 21:59:13.966869 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:13.966731 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:59:13.966869 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:13.966783 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:59:14.966604 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:14.966570 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:59:14.966993 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:14.966635 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:59:15.966727 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:15.966693 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 21:59:15.967209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:15.966794 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 21:59:16.968638 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:16.968605 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:16.970902 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:16.970878 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:16.971278 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:16.971259 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:16.971983 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:16.971957 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 21:59:17.879856 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:17.879825 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j"] Oct 30 21:59:17.883703 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:17.883683 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:17.975744 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:17.975719 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j"] Oct 30 21:59:18.042590 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042545 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042590 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042592 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042789 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042674 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042789 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042732 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042789 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042750 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042909 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042841 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042909 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042861 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042909 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042882 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.042909 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.042897 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9sm\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144083 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144120 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144139 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144170 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144156 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9sm\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144281 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144330 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144371 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144418 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144556 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144440 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144804 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144618 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144804 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144719 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144886 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144819 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.144976 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144948 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.145049 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.144954 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.146721 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.146699 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.147044 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.147022 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.201101 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.201069 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9sm\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.201554 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.201532 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.496072 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.495990 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:18.935883 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.935858 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j"] Oct 30 21:59:18.937835 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:18.937811 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8f2321_b027_4336_960d_03e9b29ff77f.slice/crio-e1a9322c3a767827ba945320a0639899fac02b63552d796d3ecd8fa2fe127ddb WatchSource:0}: Error finding container e1a9322c3a767827ba945320a0639899fac02b63552d796d3ecd8fa2fe127ddb: Status 404 returned error can't find the container with id e1a9322c3a767827ba945320a0639899fac02b63552d796d3ecd8fa2fe127ddb Oct 30 21:59:18.940150 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.940115 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:18.940230 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.940181 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:18.940230 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:18.940210 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:19.773752 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:19.773713 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" event={"ID":"fd8f2321-b027-4336-960d-03e9b29ff77f","Type":"ContainerStarted","Data":"8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5"} Oct 30 21:59:19.774161 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:19.773775 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" event={"ID":"fd8f2321-b027-4336-960d-03e9b29ff77f","Type":"ContainerStarted","Data":"e1a9322c3a767827ba945320a0639899fac02b63552d796d3ecd8fa2fe127ddb"} Oct 30 21:59:19.940530 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:19.940473 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podStartSLOduration=2.940458558 podStartE2EDuration="2.940458558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:19.939843308 +0000 UTC m=+970.129381005" watchObservedRunningTime="2026-10-30 21:59:19.940458558 +0000 UTC m=+970.129996233" Oct 30 21:59:20.499465 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:20.499425 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:20.501150 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:20.501123 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:59:20.501280 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:20.501194 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:59:21.496682 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:21.496649 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:59:21.497468 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:21.497433 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:59:21.898887 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:21.898846 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:22.499672 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:22.499639 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:59:22.500092 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:22.499720 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:59:23.499332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:23.496664 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:59:23.499332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:23.496727 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:59:24.499332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:24.496653 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:59:24.499332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:24.496714 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:59:25.497095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:25.497059 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 21:59:25.497264 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:25.497113 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 21:59:26.293947 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:26.293913 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:26.781700 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:26.781666 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:27.226351 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:27.226284 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:27.227043 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:27.227018 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:27.228035 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:27.228016 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 21:59:31.678804 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:31.678764 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:36.567553 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:36.567496 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:37.171519 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.171485 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6"] Oct 30 21:59:37.175557 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.175542 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.227209 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.227172 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0c99b1-openshift-default-dockercfg-r8hwd\"" Oct 30 21:59:37.276875 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.276843 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277024 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.276882 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277069 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277035 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277069 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277052 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277162 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277070 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277238 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277217 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mwd\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277275 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277252 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277398 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277381 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.277442 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.277410 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.278746 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.278726 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6"] Oct 30 21:59:37.378318 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378271 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378499 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378341 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378499 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378390 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378499 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378422 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378632 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378545 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378632 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378571 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378632 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378597 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378782 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378710 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mwd\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378782 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378743 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378883 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378810 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.378883 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.378860 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.379061 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.379026 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.379221 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.379201 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.379461 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.379441 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.380819 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.380799 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.381022 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.381004 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.435013 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.434959 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.435013 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.434992 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mwd\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.489775 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.489738 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:37.622207 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.622180 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v"] Oct 30 21:59:37.627091 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.627073 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.708351 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.708249 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v"] Oct 30 21:59:37.783597 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783567 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783597 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783600 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783787 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783638 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783787 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783686 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783888 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783785 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783888 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783820 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783956 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783921 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783997 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783956 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.783997 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.783977 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpflv\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.842674 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.842642 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9"] Oct 30 21:59:37.846532 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.846514 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.869655 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.869635 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6"] Oct 30 21:59:37.871663 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:37.871641 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce63103_453c_4612_b6dd_ae756c80acb2.slice/crio-ff50c2910eaf37a656f601b22a57d625282b685f5bca17bebc4de88d18115fe0 WatchSource:0}: Error finding container ff50c2910eaf37a656f601b22a57d625282b685f5bca17bebc4de88d18115fe0: Status 404 returned error can't find the container with id ff50c2910eaf37a656f601b22a57d625282b685f5bca17bebc4de88d18115fe0 Oct 30 21:59:37.873884 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.873855 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:37.873947 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.873910 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:37.873947 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.873940 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:37.885226 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885196 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885227 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885273 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885328 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885294 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885458 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885376 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885458 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885425 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885531 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885455 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpflv\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885569 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885552 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885608 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885568 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885675 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885639 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885751 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885683 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.885923 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885851 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.886000 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.885935 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.886198 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.886182 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.887754 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.887726 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.888466 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.888445 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.903376 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.903351 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" event={"ID":"cce63103-453c-4612-b6dd-ae756c80acb2","Type":"ContainerStarted","Data":"ff50c2910eaf37a656f601b22a57d625282b685f5bca17bebc4de88d18115fe0"} Oct 30 21:59:37.914339 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.910641 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9"] Oct 30 21:59:37.952434 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.952411 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.954223 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.954205 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpflv\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:37.986260 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986178 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986260 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986231 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986468 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986300 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzzn\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986468 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986416 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986468 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986461 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986622 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986497 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986622 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986579 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986622 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986610 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:37.986722 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:37.986670 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.087914 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.087887 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.087933 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.087958 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.087974 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzzn\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.087999 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088039 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088095 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088092 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088444 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088348 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088444 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088424 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088546 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088470 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088812 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088786 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088884 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088850 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.088884 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.088863 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.089057 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.089034 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.090395 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.090373 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.090783 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.090764 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.145025 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.144989 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.145297 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.145273 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzzn\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.160002 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.159975 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:38.241563 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.241437 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:38.542023 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.541994 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9"] Oct 30 21:59:38.544277 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:38.544247 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8d7813_6b35_45ff_ac01_d16abdbbf2fd.slice/crio-45372ea0653077fe6659a1225727b9385d9c1c6229e94f71c4158297ff799021 WatchSource:0}: Error finding container 45372ea0653077fe6659a1225727b9385d9c1c6229e94f71c4158297ff799021: Status 404 returned error can't find the container with id 45372ea0653077fe6659a1225727b9385d9c1c6229e94f71c4158297ff799021 Oct 30 21:59:38.546989 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.546957 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.547090 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.547019 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.547090 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.547054 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.670131 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.670102 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v"] Oct 30 21:59:38.672995 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:38.672957 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47652914_4a96_45a0_948f_5a2d262e98b5.slice/crio-8a670961127f0ded46b1d10ec5375ad8198a7bc61290c950e78f2dbb699bc910 WatchSource:0}: Error finding container 8a670961127f0ded46b1d10ec5375ad8198a7bc61290c950e78f2dbb699bc910: Status 404 returned error can't find the container with id 8a670961127f0ded46b1d10ec5375ad8198a7bc61290c950e78f2dbb699bc910 Oct 30 21:59:38.714010 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.713966 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.714152 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.714060 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.714152 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.714106 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.917065 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.917031 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" event={"ID":"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd","Type":"ContainerStarted","Data":"3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7"} Oct 30 21:59:38.917341 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.917316 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" event={"ID":"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd","Type":"ContainerStarted","Data":"45372ea0653077fe6659a1225727b9385d9c1c6229e94f71c4158297ff799021"} Oct 30 21:59:38.918972 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.918946 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" event={"ID":"cce63103-453c-4612-b6dd-ae756c80acb2","Type":"ContainerStarted","Data":"02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a"} Oct 30 21:59:38.920616 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.920597 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" event={"ID":"47652914-4a96-45a0-948f-5a2d262e98b5","Type":"ContainerStarted","Data":"28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961"} Oct 30 21:59:38.920716 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:38.920706 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" event={"ID":"47652914-4a96-45a0-948f-5a2d262e98b5","Type":"ContainerStarted","Data":"8a670961127f0ded46b1d10ec5375ad8198a7bc61290c950e78f2dbb699bc910"} Oct 30 21:59:39.040832 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.040780 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podStartSLOduration=2.040764988 podStartE2EDuration="2.040764988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.036083281 +0000 UTC m=+989.225620986" watchObservedRunningTime="2026-10-30 21:59:39.040764988 +0000 UTC m=+989.230302662" Oct 30 21:59:39.160270 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.160236 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:39.161714 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.161668 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podStartSLOduration=2.161654297 podStartE2EDuration="2.161654297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.15964943 +0000 UTC m=+989.349187134" watchObservedRunningTime="2026-10-30 21:59:39.161654297 +0000 UTC m=+989.351191972" Oct 30 21:59:39.161714 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.161696 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:39.161926 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.161757 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:39.241997 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.241962 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:39.244493 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.244468 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:39.244604 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.244521 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:39.282332 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.281883 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podStartSLOduration=2.281864985 podStartE2EDuration="2.281864985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.275399707 +0000 UTC m=+989.464937394" watchObservedRunningTime="2026-10-30 21:59:39.281864985 +0000 UTC m=+989.471402660" Oct 30 21:59:39.497392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.490717 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:39.497392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.493269 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:59:39.497392 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:39.493358 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 21:59:40.163779 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.163733 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:40.164191 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.163838 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:40.252404 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.243660 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:40.252404 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.243731 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:40.455582 ip-10-0-134-73 kubenswrapper[5508]: E1030 21:59:40.455496 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 21:59:40.492784 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.492745 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:59:40.493079 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.493050 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 21:59:40.704330 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:40.704267 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:41.160893 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:41.160859 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.161080 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:41.160919 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:41.248863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:41.248721 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.248863 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:41.248808 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:41.490799 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:41.490706 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.491064 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:41.491034 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 21:59:42.026397 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.026223 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:42.166366 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.162723 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:42.166366 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.162788 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:42.249691 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.249654 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:42.250281 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.250253 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:42.490749 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.490714 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:59:42.491356 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:42.491294 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 21:59:43.160897 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:43.160853 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:43.161106 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:43.160914 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:43.250333 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:43.247734 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:43.250333 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:43.247803 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:43.511904 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:43.511809 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:59:43.515432 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:43.515390 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 21:59:44.160451 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:44.160420 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:44.160628 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:44.160469 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:44.243838 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:44.243805 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:44.244094 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:44.244067 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:44.492387 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:44.491743 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 21:59:44.492387 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:44.491809 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 21:59:45.161565 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.161515 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:59:45.161782 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.161595 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:59:45.242225 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.242186 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:59:45.242471 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.242268 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:59:45.496492 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.496420 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:45.496973 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.496954 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:45.497781 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:45.497756 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 21:59:46.164150 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:46.164122 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:46.164673 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:46.164645 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:46.165427 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:46.165409 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 21:59:46.246103 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:46.246072 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:46.246434 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:46.246415 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:46.247042 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:46.247021 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 21:59:47.717073 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:47.717037 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:52.574292 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:52.574217 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:56.426168 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.426134 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz"] Oct 30 21:59:56.429900 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.429883 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.502050 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.502014 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.502200 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.502145 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.502200 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.502181 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.502329 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.502295 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.502387 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.502333 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvwb\" (UniqueName: \"kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.502445 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.502422 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.531137 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.531116 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"default-dockercfg-sdlj2\"" Oct 30 21:59:56.531137 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.531119 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"distinct-w4-b0c55f-kserve-self-signed-certs\"" Oct 30 21:59:56.531358 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.531322 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"kube-root-ca.crt\"" Oct 30 21:59:56.531421 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.531389 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"openshift-service-ca.crt\"" Oct 30 21:59:56.533781 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.533758 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz"] Oct 30 21:59:56.551259 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.551232 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:56.603655 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.603618 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.603900 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.603671 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvwb\" (UniqueName: \"kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.603900 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.603711 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.603900 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.603846 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.608376 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.603957 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.609036 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.608432 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.609036 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.608913 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.609036 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.607888 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.609252 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.609169 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.610591 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.610566 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.610890 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.610853 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.665032 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.665001 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvwb\" (UniqueName: \"kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb\") pod \"distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:56.741901 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:56.741830 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 21:59:57.121593 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:57.121486 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz"] Oct 30 21:59:57.123833 ip-10-0-134-73 kubenswrapper[5508]: W1030 21:59:57.123806 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc09539f_7ac0_4241_b29a_cd19a9cd5f96.slice/crio-6afbc0b2b97aead7e64b7c594d3c766675406281ca03e59baf071617b1764c87 WatchSource:0}: Error finding container 6afbc0b2b97aead7e64b7c594d3c766675406281ca03e59baf071617b1764c87: Status 404 returned error can't find the container with id 6afbc0b2b97aead7e64b7c594d3c766675406281ca03e59baf071617b1764c87 Oct 30 21:59:57.517768 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:57.517684 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:58.108945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:58.108908 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" event={"ID":"bc09539f-7ac0-4241-b29a-cd19a9cd5f96","Type":"ContainerStarted","Data":"3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404"} Oct 30 21:59:58.108945 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:58.108946 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" event={"ID":"bc09539f-7ac0-4241-b29a-cd19a9cd5f96","Type":"ContainerStarted","Data":"6afbc0b2b97aead7e64b7c594d3c766675406281ca03e59baf071617b1764c87"} Oct 30 21:59:58.227192 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:58.227143 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podStartSLOduration=2.227130505 podStartE2EDuration="2.227130505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:58.222943537 +0000 UTC m=+1008.412481212" watchObservedRunningTime="2026-10-30 21:59:58.227130505 +0000 UTC m=+1008.416668180" Oct 30 21:59:58.742041 ip-10-0-134-73 kubenswrapper[5508]: I1030 21:59:58.742009 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:00:00.214775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.214737 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5"] Oct 30 22:00:00.219005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.218981 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.333292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.333254 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5"] Oct 30 22:00:00.338792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.338752 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:00:00.351408 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.351375 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7r66\" (UniqueName: \"kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.351563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.351494 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.452583 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.452550 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7r66\" (UniqueName: \"kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.452767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.452611 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.453195 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.453175 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.509753 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.509687 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7r66\" (UniqueName: \"kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.533731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.533709 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:00.912114 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.912091 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5"] Oct 30 22:00:00.914482 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:00:00.914445 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd9122f_9045_4ff2_ab2f_9688da8c89cb.slice/crio-742b028f2308017329356ec71d5d4f88c44d2ce97e3207025daaa4b43b437d49 WatchSource:0}: Error finding container 742b028f2308017329356ec71d5d4f88c44d2ce97e3207025daaa4b43b437d49: Status 404 returned error can't find the container with id 742b028f2308017329356ec71d5d4f88c44d2ce97e3207025daaa4b43b437d49 Oct 30 22:00:00.916485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:00.916465 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:00:01.126751 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:01.126710 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" event={"ID":"7dd9122f-9045-4ff2-ab2f-9688da8c89cb","Type":"ContainerStarted","Data":"742b028f2308017329356ec71d5d4f88c44d2ce97e3207025daaa4b43b437d49"} Oct 30 22:00:02.365811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:02.365769 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:06.157593 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:06.157505 5508 generic.go:356] "Generic (PLEG): container finished" podID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerID="d03471d2b1bf7f8cd04970166e74ef40c29f29dc3848e998d0f8ba8dee579b74" exitCode=60 Oct 30 22:00:06.158034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:06.157587 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" event={"ID":"7dd9122f-9045-4ff2-ab2f-9688da8c89cb","Type":"ContainerDied","Data":"d03471d2b1bf7f8cd04970166e74ef40c29f29dc3848e998d0f8ba8dee579b74"} Oct 30 22:00:06.158034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:06.157905 5508 scope.go:122] "RemoveContainer" containerID="d03471d2b1bf7f8cd04970166e74ef40c29f29dc3848e998d0f8ba8dee579b74" Oct 30 22:00:06.743497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:06.742747 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:00:06.761124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:06.761091 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:00:07.168319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:07.168280 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" event={"ID":"7dd9122f-9045-4ff2-ab2f-9688da8c89cb","Type":"ContainerStarted","Data":"ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a"} Oct 30 22:00:07.181544 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:07.181508 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:00:07.285071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:07.284998 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" podStartSLOduration=3.093398606 podStartE2EDuration="7.28497978s" totalImagesPullingTime="4.191581174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:07.281490593 +0000 UTC m=+1017.471028269" watchObservedRunningTime="2026-10-30 22:00:07.28497978 +0000 UTC m=+1017.474517456" Oct 30 22:00:07.351935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:07.351828 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.176040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:08.175917 5508 generic.go:356] "Generic (PLEG): container finished" podID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerID="ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a" exitCode=60 Oct 30 22:00:08.176040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:08.175959 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" event={"ID":"7dd9122f-9045-4ff2-ab2f-9688da8c89cb","Type":"ContainerDied","Data":"ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a"} Oct 30 22:00:08.176040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:08.176040 5508 scope.go:122] "RemoveContainer" containerID="d03471d2b1bf7f8cd04970166e74ef40c29f29dc3848e998d0f8ba8dee579b74" Oct 30 22:00:08.200196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:08.176348 5508 scope.go:122] "RemoveContainer" containerID="ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a" Oct 30 22:00:08.200196 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:00:08.176662 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5_odh-ai-gateway-infra(7dd9122f-9045-4ff2-ab2f-9688da8c89cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" Oct 30 22:00:09.184603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:09.184572 5508 scope.go:122] "RemoveContainer" containerID="ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a" Oct 30 22:00:09.184984 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:00:09.184843 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5_odh-ai-gateway-infra(7dd9122f-9045-4ff2-ab2f-9688da8c89cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" Oct 30 22:00:10.375409 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:00:10.375367 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:00:12.291848 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:12.291817 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:12.670958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:12.670923 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:12.953169 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:12.953079 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:17.182043 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:17.182003 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:18.945560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:18.945530 5508 scope.go:122] "RemoveContainer" containerID="ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a" Oct 30 22:00:20.258506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:20.258466 5508 generic.go:356] "Generic (PLEG): container finished" podID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerID="d50b9ddfd4cc6f119bccc47ebea9effbb024c5b7c48a3f37661d03b6a33ef4bc" exitCode=60 Oct 30 22:00:20.258952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:20.258533 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" event={"ID":"7dd9122f-9045-4ff2-ab2f-9688da8c89cb","Type":"ContainerDied","Data":"d50b9ddfd4cc6f119bccc47ebea9effbb024c5b7c48a3f37661d03b6a33ef4bc"} Oct 30 22:00:20.258952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:20.258606 5508 scope.go:122] "RemoveContainer" containerID="ce6bcfb89911db5bccbd3abdaa9b46aee6e37ac007975b4e56a659aa2845b81a" Oct 30 22:00:20.259089 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:20.259072 5508 scope.go:122] "RemoveContainer" containerID="d50b9ddfd4cc6f119bccc47ebea9effbb024c5b7c48a3f37661d03b6a33ef4bc" Oct 30 22:00:20.259500 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:00:20.259470 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5_odh-ai-gateway-infra(7dd9122f-9045-4ff2-ab2f-9688da8c89cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" Oct 30 22:00:21.383602 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.383573 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5"] Oct 30 22:00:21.539552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.539524 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:21.618980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.618948 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66\" (UniqueName: \"kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66\") pod \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " Oct 30 22:00:21.619154 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.619049 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca\") pod \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\" (UID: \"7dd9122f-9045-4ff2-ab2f-9688da8c89cb\") " Oct 30 22:00:21.619403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.619379 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca" pod "7dd9122f-9045-4ff2-ab2f-9688da8c89cb" (UID: "7dd9122f-9045-4ff2-ab2f-9688da8c89cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.619519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.619505 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.621365 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.621336 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66" pod "7dd9122f-9045-4ff2-ab2f-9688da8c89cb" (UID: "7dd9122f-9045-4ff2-ab2f-9688da8c89cb"). InnerVolumeSpecName "kube-api-access-x7r66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.720793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:21.720713 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7r66\" (UniqueName: \"kubernetes.io/projected/7dd9122f-9045-4ff2-ab2f-9688da8c89cb-kube-api-access-x7r66\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:22.092396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.092363 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:22.253003 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.252956 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:22.279171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.279139 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" Oct 30 22:00:22.279410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.279132 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5" event={"ID":"7dd9122f-9045-4ff2-ab2f-9688da8c89cb","Type":"ContainerDied","Data":"742b028f2308017329356ec71d5d4f88c44d2ce97e3207025daaa4b43b437d49"} Oct 30 22:00:22.279565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.279542 5508 scope.go:122] "RemoveContainer" containerID="d50b9ddfd4cc6f119bccc47ebea9effbb024c5b7c48a3f37661d03b6a33ef4bc" Oct 30 22:00:22.442809 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.442721 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5"] Oct 30 22:00:22.447993 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:22.447967 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889960-775f5"] Oct 30 22:00:23.953318 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:23.953284 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" path="/var/lib/kubelet/pods/7dd9122f-9045-4ff2-ab2f-9688da8c89cb/volumes" Oct 30 22:00:26.643983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:26.643942 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:27.096116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:27.096081 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:27.648947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:27.648910 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:27.816827 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:27.816792 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:31.881381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:31.881340 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:36.662598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.662566 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:36.745931 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.745898 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh"] Oct 30 22:00:36.746581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.746559 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:00:36.746581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.746580 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:00:36.746768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.746673 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:00:36.746768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.746682 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:00:36.746863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.746800 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:00:36.746863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.746814 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:00:36.754435 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.754412 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.802701 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.802665 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.802872 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.802852 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.802929 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.802911 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6j4n\" (UniqueName: \"kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.803023 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.803005 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.803087 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.803058 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.803181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.803164 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.807092 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.807065 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh"] Oct 30 22:00:36.853380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.853342 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"openshift-service-ca.crt\"" Oct 30 22:00:36.853556 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.853426 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"distinct-w6-29cbad-kserve-self-signed-certs\"" Oct 30 22:00:36.853556 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.853504 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"default-dockercfg-w89h2\"" Oct 30 22:00:36.855538 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.855517 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"kube-root-ca.crt\"" Oct 30 22:00:36.872626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.872596 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:36.904645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.904609 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.905434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.905294 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.905580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.905441 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6j4n\" (UniqueName: \"kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.905580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.905518 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.905580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.905573 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.905774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.905674 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.905774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.905728 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.906284 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.906258 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.908926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.908897 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.911470 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.911445 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.917252 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.917161 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:36.963091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:36.963057 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6j4n\" (UniqueName: \"kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n\") pod \"distinct-w6-29cbad-kserve-6cd8b769db-wt7vh\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:37.067282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:37.067246 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:37.456410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:37.456373 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh"] Oct 30 22:00:37.458893 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:00:37.458867 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b9d2e45_3c03_4a9d_aa6c_6f1c3a50e0da.slice/crio-6a744ca99fd9b767148526962ddfb609455e17212904f6442bf4878aa7b1aea0 WatchSource:0}: Error finding container 6a744ca99fd9b767148526962ddfb609455e17212904f6442bf4878aa7b1aea0: Status 404 returned error can't find the container with id 6a744ca99fd9b767148526962ddfb609455e17212904f6442bf4878aa7b1aea0 Oct 30 22:00:38.385683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:38.385646 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" event={"ID":"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da","Type":"ContainerStarted","Data":"08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65"} Oct 30 22:00:38.386105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:38.385700 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" event={"ID":"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da","Type":"ContainerStarted","Data":"6a744ca99fd9b767148526962ddfb609455e17212904f6442bf4878aa7b1aea0"} Oct 30 22:00:38.500561 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:38.500494 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podStartSLOduration=2.500477981 podStartE2EDuration="2.500477981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:38.499359314 +0000 UTC m=+1048.688896998" watchObservedRunningTime="2026-10-30 22:00:38.500477981 +0000 UTC m=+1048.690015655" Oct 30 22:00:39.067620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:39.067590 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:40.373054 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:00:40.373012 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:00:41.569372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:41.569267 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:46.656000 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:46.655964 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:47.067710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:47.067616 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:47.080611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:47.080566 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:47.467717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:47.467690 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:00:51.562135 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:51.562101 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:56.376285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:00:56.376247 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.727959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:00.727922 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.272182 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:01.272146 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:02.938522 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:02.938488 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:06.170536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:06.170505 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:08.834545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.834505 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8"] Oct 30 22:01:08.835073 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.835058 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:01:08.835126 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.835077 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:01:08.835196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.835185 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd9122f-9045-4ff2-ab2f-9688da8c89cb" containerName="cleanup" Oct 30 22:01:08.838777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.838761 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.887516 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.887477 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44"] Oct 30 22:01:08.891177 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.891150 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.891354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.891221 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxw7\" (UniqueName: \"kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.891354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.891347 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.891477 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.891435 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.891571 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.891549 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.891629 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.891601 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.892152 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.892137 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.932735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.932674 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"distinct-w2-0c99b1-kserve-self-signed-certs\"" Oct 30 22:01:08.932881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.932843 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"default-dockercfg-cp7k7\"" Oct 30 22:01:08.932944 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.932902 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"kube-root-ca.crt\"" Oct 30 22:01:08.948558 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.948533 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"openshift-service-ca.crt\"" Oct 30 22:01:08.951012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.950991 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8"] Oct 30 22:01:08.986408 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.986377 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44"] Oct 30 22:01:08.996001 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.995966 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.996372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.996346 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.996668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.996609 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.996759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.996680 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nhq\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.996836 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.996758 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.996934 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.996838 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.996998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.996949 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.997085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997040 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.997234 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997170 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.997486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997264 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.997486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997345 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.997486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997425 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.997486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997470 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:08.999877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997579 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxw7\" (UniqueName: \"kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.999877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.997629 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.999877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.999034 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:08.999877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:08.999355 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:09.005923 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.004460 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:09.016111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.016078 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.022281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.022247 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:09.027707 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.027657 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:09.062212 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.062185 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxw7\" (UniqueName: \"kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7\") pod \"distinct-w2-0c99b1-kserve-74b7bd557c-w78b8\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:09.098935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.098869 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.098935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.098907 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.098935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.098936 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099026 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099061 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099088 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nhq\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099129 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099208 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099255 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099372 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099391 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099592 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099576 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099681 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.099941 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.099922 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.101315 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.101283 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.101681 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.101664 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.151480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.151452 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:09.156142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.155930 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.156142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.155992 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nhq\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.206407 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.206379 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:09.254980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.254571 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2"] Oct 30 22:01:09.265288 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.264656 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.310768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.310728 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.310952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.310795 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.310952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.310850 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.310952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.310939 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.311134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.310997 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.311134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.311012 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.311134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.311033 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqlw\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.311134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.311062 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.311134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.311093 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.326737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.326682 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2"] Oct 30 22:01:09.412397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412323 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412418 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412445 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412478 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqlw\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412520 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412567 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412617 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412673 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412728 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412941 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412800 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412941 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412839 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.412941 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412884 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.413027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.412953 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.413282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.413259 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.414892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.414874 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.415156 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.415137 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.470104 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.470041 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.475786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.475757 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqlw\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.528658 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.528634 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8"] Oct 30 22:01:09.530247 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:01:09.530221 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb090b2_8a3c_49a3_8c06_13f907d41fa2.slice/crio-013f83ad57a7c3a3aa109fd74aface10df3388a3854cc133c723e821277f1217 WatchSource:0}: Error finding container 013f83ad57a7c3a3aa109fd74aface10df3388a3854cc133c723e821277f1217: Status 404 returned error can't find the container with id 013f83ad57a7c3a3aa109fd74aface10df3388a3854cc133c723e821277f1217 Oct 30 22:01:09.581935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.581902 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" event={"ID":"3fb090b2-8a3c-49a3-8c06-13f907d41fa2","Type":"ContainerStarted","Data":"013f83ad57a7c3a3aa109fd74aface10df3388a3854cc133c723e821277f1217"} Oct 30 22:01:09.589630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.589604 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:09.668602 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.668463 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44"] Oct 30 22:01:09.678947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.677898 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:09.678947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.677994 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:09.678947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.678035 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:09.989037 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:01:09.989007 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8677b9d5_ec7b_4b07_bfdb_585af3c3f40c.slice/crio-bbfe48bf2cf164be3d33a84cf9a4cb1706873fbc1af0ba4cb5a5b6e41d77b24b WatchSource:0}: Error finding container bbfe48bf2cf164be3d33a84cf9a4cb1706873fbc1af0ba4cb5a5b6e41d77b24b: Status 404 returned error can't find the container with id bbfe48bf2cf164be3d33a84cf9a4cb1706873fbc1af0ba4cb5a5b6e41d77b24b Oct 30 22:01:09.989380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.989040 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2"] Oct 30 22:01:09.991184 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.991145 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:09.991283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.991226 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:09.991283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:09.991269 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:10.374816 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:01:10.374746 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:01:10.606329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.605342 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" event={"ID":"3fb090b2-8a3c-49a3-8c06-13f907d41fa2","Type":"ContainerStarted","Data":"ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f"} Oct 30 22:01:10.610325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.608606 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" event={"ID":"30a1d1b2-c64b-4775-ac0c-115289b715d8","Type":"ContainerStarted","Data":"cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d"} Oct 30 22:01:10.610325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.608664 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" event={"ID":"30a1d1b2-c64b-4775-ac0c-115289b715d8","Type":"ContainerStarted","Data":"786633a1772991b5f366854d44e51a53f56129b70b8c6d36bbf6445d92e3c5c8"} Oct 30 22:01:10.628790 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.628699 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" event={"ID":"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c","Type":"ContainerStarted","Data":"093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69"} Oct 30 22:01:10.628790 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.628766 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" event={"ID":"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c","Type":"ContainerStarted","Data":"bbfe48bf2cf164be3d33a84cf9a4cb1706873fbc1af0ba4cb5a5b6e41d77b24b"} Oct 30 22:01:10.722102 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.721574 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podStartSLOduration=2.7215566239999998 podStartE2EDuration="2.721556624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:10.719192749 +0000 UTC m=+1080.908730425" watchObservedRunningTime="2026-10-30 22:01:10.721556624 +0000 UTC m=+1080.911094299" Oct 30 22:01:10.837957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.837899 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podStartSLOduration=2.837885392 podStartE2EDuration="2.837885392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:10.835474926 +0000 UTC m=+1081.025012614" watchObservedRunningTime="2026-10-30 22:01:10.837885392 +0000 UTC m=+1081.027423067" Oct 30 22:01:10.959116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:10.959012 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podStartSLOduration=1.9589972759999998 podStartE2EDuration="1.958997276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:10.956408004 +0000 UTC m=+1081.145945712" watchObservedRunningTime="2026-10-30 22:01:10.958997276 +0000 UTC m=+1081.148534956" Oct 30 22:01:11.144393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.144349 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:11.151750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.151719 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:11.226777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.226692 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:11.236382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.234143 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:01:11.236382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.234202 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:01:11.595331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.591364 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:11.595331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.592438 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:01:11.595331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:11.592490 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:01:12.208792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:12.208754 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:01:12.209433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:12.209397 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:01:12.601332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:12.590643 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:01:12.601332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:12.590706 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:01:13.214331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:13.210671 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:01:13.214331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:13.210736 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:01:13.596331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:13.595424 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.596331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:13.595958 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:01:13.596331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:13.596021 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:01:14.210331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:14.209717 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:01:14.210331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:14.209778 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:01:14.601330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:14.600474 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:01:14.601330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:14.600535 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:01:15.211330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:15.210788 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:01:15.211330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:15.210852 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:01:15.593741 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:15.593702 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:01:15.594429 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:15.594396 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:01:16.207230 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:16.207188 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:01:16.207750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:16.207257 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 22:01:16.541565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:16.541473 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:16.590970 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:16.590857 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:01:16.590970 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:16.590930 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:01:17.211098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:17.211068 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:17.211583 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:17.211565 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:17.212209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:17.212187 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:01:17.595818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:17.595786 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:17.596454 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:17.596424 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:17.597226 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:17.597201 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:01:19.151785 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:19.151746 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:19.164212 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:19.164192 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:19.707097 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:19.707066 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:01:21.371879 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:21.371843 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.183920 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:26.183886 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.992629 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:26.992592 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.048168 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:31.048132 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.521636 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:31.521602 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.674271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:31.674238 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:36.072056 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:36.072006 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:40.372678 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:01:40.372641 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:01:40.872930 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:40.872889 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:45.888747 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:45.888711 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:50.750708 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:50.750675 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:54.611770 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.611731 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl"] Oct 30 22:01:54.615585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.615566 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.719121 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.719093 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"distinct-2-w4-b0c55f-kserve-self-signed-certs\"" Oct 30 22:01:54.721847 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.721819 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl"] Oct 30 22:01:54.724721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.724688 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:54.753368 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.753344 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.753527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.753413 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4k9\" (UniqueName: \"kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.753594 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.753520 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.753651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.753634 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.753709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.753697 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.753764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.753731 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855107 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855169 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855274 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855359 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4k9\" (UniqueName: \"kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855407 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855482 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855611 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855725 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855701 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.855799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.855782 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.857764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.857739 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.857917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.857897 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.912167 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.912102 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4k9\" (UniqueName: \"kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9\") pod \"distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:54.928279 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:54.928250 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:01:55.306348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:55.306321 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl"] Oct 30 22:01:55.308488 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:01:55.308456 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87c637bc_9f13_4e04_b988_a07c04ee163d.slice/crio-2e445483fec7d27f0147bec8bfe8fe6b925761755cb5ad9313aec05dffe1c5f1 WatchSource:0}: Error finding container 2e445483fec7d27f0147bec8bfe8fe6b925761755cb5ad9313aec05dffe1c5f1: Status 404 returned error can't find the container with id 2e445483fec7d27f0147bec8bfe8fe6b925761755cb5ad9313aec05dffe1c5f1 Oct 30 22:01:55.697513 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:55.697482 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:55.940567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:55.940528 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" event={"ID":"87c637bc-9f13-4e04-b988-a07c04ee163d","Type":"ContainerStarted","Data":"31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7"} Oct 30 22:01:55.940751 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:55.940581 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" event={"ID":"87c637bc-9f13-4e04-b988-a07c04ee163d","Type":"ContainerStarted","Data":"2e445483fec7d27f0147bec8bfe8fe6b925761755cb5ad9313aec05dffe1c5f1"} Oct 30 22:01:56.057296 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:56.057185 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podStartSLOduration=2.057167312 podStartE2EDuration="2.057167312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:56.056033915 +0000 UTC m=+1126.245571591" watchObservedRunningTime="2026-10-30 22:01:56.057167312 +0000 UTC m=+1126.246705009" Oct 30 22:01:56.928891 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:01:56.928861 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:02:00.479557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:00.479483 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:04.928903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:04.928875 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:02:04.942693 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:04.942668 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:02:05.004577 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:05.004548 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:02:05.505668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:05.505631 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:10.372852 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:02:10.372813 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:02:10.412630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:10.412601 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.944092 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:14.944045 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:15.391985 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:15.391944 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:20.303962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:20.303930 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.866116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:23.866081 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29"] Oct 30 22:02:23.870451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:23.870428 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:23.915204 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:23.915173 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4"] Oct 30 22:02:23.918983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:23.918964 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.012991 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013205 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013047 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013284 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013195 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013284 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013236 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013367 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.013451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013418 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.013451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013444 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.013615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013477 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxf6\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.013615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013511 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013540 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013566 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013634 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.013819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013665 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.013819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013706 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.013819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013747 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.014025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013920 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.014025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013951 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.014025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.013976 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskrc\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.023537 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.023508 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4"] Oct 30 22:02:24.080108 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.080077 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29"] Oct 30 22:02:24.115811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.115760 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.115913 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.115944 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116008 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116064 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116089 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116124 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxf6\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116156 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116190 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116216 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116325 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116361 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116403 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116445 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116571 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116574 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116630 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.116849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116656 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskrc\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.116849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116698 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.117274 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.117052 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.117374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.117350 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.117606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.116217 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.117760 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.117736 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.117889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.117870 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.118054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.118032 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.118054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.118050 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.119721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.118768 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.119721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.119409 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.119721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.119643 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.120123 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.120099 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.120296 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.120278 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.122218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.122196 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.174165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.174136 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.174324 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.174188 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskrc\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.174324 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.174241 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxf6\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.174505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.174483 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.183729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.183533 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:24.230816 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.230781 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:24.615770 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.615725 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29"] Oct 30 22:02:24.657405 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:02:24.657374 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f010838_ed32_44bc_aa3d_f59b6400be81.slice/crio-dfe92aee550bb4370a36d7daef9a04a1862740e6202499ffb89ecf562085c019 WatchSource:0}: Error finding container dfe92aee550bb4370a36d7daef9a04a1862740e6202499ffb89ecf562085c019: Status 404 returned error can't find the container with id dfe92aee550bb4370a36d7daef9a04a1862740e6202499ffb89ecf562085c019 Oct 30 22:02:24.659845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.659808 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.659938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.659878 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.659938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.659905 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.669757 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.669733 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4"] Oct 30 22:02:24.671280 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:02:24.671257 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37368dad_c429_479b_a803_4d4429f6bb44.slice/crio-cdc807d4b092ae536239be024ae949e87c371279251ce87266668f1c0823313a WatchSource:0}: Error finding container cdc807d4b092ae536239be024ae949e87c371279251ce87266668f1c0823313a: Status 404 returned error can't find the container with id cdc807d4b092ae536239be024ae949e87c371279251ce87266668f1c0823313a Oct 30 22:02:24.674601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.673852 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.674601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.673944 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.674601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:24.673990 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:25.127041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.126989 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" event={"ID":"37368dad-c429-479b-a803-4d4429f6bb44","Type":"ContainerStarted","Data":"b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee"} Oct 30 22:02:25.127565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.127098 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" event={"ID":"37368dad-c429-479b-a803-4d4429f6bb44","Type":"ContainerStarted","Data":"cdc807d4b092ae536239be024ae949e87c371279251ce87266668f1c0823313a"} Oct 30 22:02:25.133954 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.133929 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" event={"ID":"7f010838-ed32-44bc-aa3d-f59b6400be81","Type":"ContainerStarted","Data":"3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918"} Oct 30 22:02:25.134087 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.133974 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" event={"ID":"7f010838-ed32-44bc-aa3d-f59b6400be81","Type":"ContainerStarted","Data":"dfe92aee550bb4370a36d7daef9a04a1862740e6202499ffb89ecf562085c019"} Oct 30 22:02:25.184735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.184708 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:25.186022 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.185991 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:25.186123 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.186035 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:25.205462 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.205433 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:25.232514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.232040 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:25.240476 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.240447 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:25.240653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.240502 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:25.251590 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.251525 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podStartSLOduration=2.251505566 podStartE2EDuration="2.251505566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:25.246152526 +0000 UTC m=+1155.435690197" watchObservedRunningTime="2026-10-30 22:02:25.251505566 +0000 UTC m=+1155.441043243" Oct 30 22:02:25.363381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:25.363326 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podStartSLOduration=2.363292957 podStartE2EDuration="2.363292957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:25.361414668 +0000 UTC m=+1155.550952368" watchObservedRunningTime="2026-10-30 22:02:25.363292957 +0000 UTC m=+1155.552830633" Oct 30 22:02:26.194329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:26.186732 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:26.194329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:26.186795 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:26.231480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:26.231445 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:26.231654 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:26.231498 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:27.185098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:27.185063 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:27.185418 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:27.185389 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:27.231656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:27.231627 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:27.232103 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:27.232077 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:27.329504 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:27.329201 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:28.187697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:28.187662 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.187981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:28.187951 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:28.231372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:28.231336 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.231544 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:28.231396 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:29.202329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:29.201699 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.202329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:29.201769 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:29.232730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:29.232695 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.232951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:29.232928 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:30.151822 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:30.151766 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:30.188891 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:30.188852 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.189280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:30.189249 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:30.239330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:30.233727 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.239330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:30.233788 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:31.184871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:31.184835 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:02:31.185062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:31.184893 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 22:02:31.231849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:31.231816 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:02:31.231992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:31.231883 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 22:02:32.188954 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:32.188921 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:32.189708 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:32.189682 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:32.190701 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:32.190680 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:02:32.361321 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:32.361279 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:32.361660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:32.361640 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:32.362440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:32.362418 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:02:34.012257 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:34.012221 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:35.189618 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:35.189577 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:40.081184 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:40.081151 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:40.373617 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:02:40.373542 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:02:45.019031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:45.018985 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:47.455403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:47.455356 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:50.079919 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:50.079837 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.264554 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.264522 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc"] Oct 30 22:02:54.269803 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.269782 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.354710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.354679 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.354861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.354715 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.354861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.354735 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.354936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.354898 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccjt\" (UniqueName: \"kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.354990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.354972 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.355044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.355024 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.372729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.372706 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"distinct-2-w6-29cbad-kserve-self-signed-certs\"" Oct 30 22:02:54.375468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.375447 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc"] Oct 30 22:02:54.388904 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.388871 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.456756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.456722 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.456900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.456772 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.456900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.456802 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.456992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.456904 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccjt\" (UniqueName: \"kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.456992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.456981 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.457057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.457020 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.457543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.457518 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.460284 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.460261 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.461378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.461356 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.469200 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.469170 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.476862 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.476833 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.516574 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.516492 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccjt\" (UniqueName: \"kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt\") pod \"distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.582536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.582476 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:54.960146 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:02:54.960119 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafcc5a28_a808_4041_99a3_c27b496b8bb3.slice/crio-1faab68a45d731d7ec58703290ae0a03675b25ec695e8772ec1d8ac4b9e7fc52 WatchSource:0}: Error finding container 1faab68a45d731d7ec58703290ae0a03675b25ec695e8772ec1d8ac4b9e7fc52: Status 404 returned error can't find the container with id 1faab68a45d731d7ec58703290ae0a03675b25ec695e8772ec1d8ac4b9e7fc52 Oct 30 22:02:54.960344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.960317 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc"] Oct 30 22:02:54.992706 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:54.992677 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.348871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:55.348833 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" event={"ID":"afcc5a28-a808-4041-99a3-c27b496b8bb3","Type":"ContainerStarted","Data":"0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea"} Oct 30 22:02:55.349334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:55.348882 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" event={"ID":"afcc5a28-a808-4041-99a3-c27b496b8bb3","Type":"ContainerStarted","Data":"1faab68a45d731d7ec58703290ae0a03675b25ec695e8772ec1d8ac4b9e7fc52"} Oct 30 22:02:55.465538 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:55.465481 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podStartSLOduration=1.465462655 podStartE2EDuration="1.465462655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:55.462256441 +0000 UTC m=+1185.651794137" watchObservedRunningTime="2026-10-30 22:02:55.465462655 +0000 UTC m=+1185.655000335" Oct 30 22:02:55.583505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:55.583472 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:02:59.879008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:02:59.878975 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:00.645373 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:00.645332 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:04.583637 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:04.583608 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:03:04.596540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:04.596517 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:03:04.948683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:04.948604 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.429003 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:05.428975 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:03:09.435281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.435248 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx"] Oct 30 22:03:09.440716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.440694 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.484670 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.484637 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz"] Oct 30 22:03:09.488706 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.488684 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.562993 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.562967 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563163 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563024 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563163 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563133 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.563276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563180 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563218 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.563426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563288 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmd89\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563386 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563447 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.563521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563486 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.563626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563578 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563628 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563700 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.563806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563790 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.563857 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563822 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295dh\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.563934 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563914 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.564017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.563998 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.564082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.564030 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.564082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.564059 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.589690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.589666 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz"] Oct 30 22:03:09.646294 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.646261 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx"] Oct 30 22:03:09.665010 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.664968 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665010 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665011 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665035 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665090 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665134 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665169 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665203 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665238 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665271 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmd89\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665342 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665389 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665421 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665466 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665494 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665519 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665566 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665583 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.665611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665599 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.666006 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665620 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.666006 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.665630 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295dh\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.666102 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.666031 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.667266 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.667242 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.667437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.667368 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.667628 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.667602 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.668176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.668153 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.668369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.668349 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.668769 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.668748 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.669570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.669447 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.675551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.671208 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.675551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.671962 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.675551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.673349 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.683904 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.683882 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.722883 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.722827 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.723821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.723799 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.724185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.724167 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295dh\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.724241 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.724225 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmd89\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:09.755484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.755462 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:09.779218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.779191 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:09.802180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:09.802153 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:10.189228 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.189203 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx"] Oct 30 22:03:10.190260 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:03:10.190242 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0527abff_b56e_457a_a62f_cae88f284eea.slice/crio-08dd7d09b7bbf4a2c4c2845f0bf33f1aecabbb2002589a01b819a468a63a258f WatchSource:0}: Error finding container 08dd7d09b7bbf4a2c4c2845f0bf33f1aecabbb2002589a01b819a468a63a258f: Status 404 returned error can't find the container with id 08dd7d09b7bbf4a2c4c2845f0bf33f1aecabbb2002589a01b819a468a63a258f Oct 30 22:03:10.192597 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.192564 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.192690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.192626 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.192690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.192655 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.246834 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.246809 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz"] Oct 30 22:03:10.251165 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:03:10.251142 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37a9dba_9477_4e36_9e10_422bf6286e1b.slice/crio-e17ea7b1d0c5fcee461053a82e4f4a6b9401fd1eb96d85c7003163b3c751384c WatchSource:0}: Error finding container e17ea7b1d0c5fcee461053a82e4f4a6b9401fd1eb96d85c7003163b3c751384c: Status 404 returned error can't find the container with id e17ea7b1d0c5fcee461053a82e4f4a6b9401fd1eb96d85c7003163b3c751384c Oct 30 22:03:10.253628 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.253595 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.253725 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.253673 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.253788 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.253736 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.355136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.355101 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.377610 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:03:10.377582 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:03:10.455248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.455162 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" event={"ID":"0527abff-b56e-457a-a62f-cae88f284eea","Type":"ContainerStarted","Data":"0954ee3127d8945a33df82e25d65e9edcb8d41babcdfa55153bff38ffe3a4920"} Oct 30 22:03:10.455248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.455227 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" event={"ID":"0527abff-b56e-457a-a62f-cae88f284eea","Type":"ContainerStarted","Data":"08dd7d09b7bbf4a2c4c2845f0bf33f1aecabbb2002589a01b819a468a63a258f"} Oct 30 22:03:10.457251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.457171 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" event={"ID":"e37a9dba-9477-4e36-9e10-422bf6286e1b","Type":"ContainerStarted","Data":"3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807"} Oct 30 22:03:10.457251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.457217 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" event={"ID":"e37a9dba-9477-4e36-9e10-422bf6286e1b","Type":"ContainerStarted","Data":"e17ea7b1d0c5fcee461053a82e4f4a6b9401fd1eb96d85c7003163b3c751384c"} Oct 30 22:03:10.573785 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.573720 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podStartSLOduration=1.573700339 podStartE2EDuration="1.573700339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:10.572956796 +0000 UTC m=+1200.762494472" watchObservedRunningTime="2026-10-30 22:03:10.573700339 +0000 UTC m=+1200.763238015" Oct 30 22:03:10.699104 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.699038 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podStartSLOduration=1.69901792 podStartE2EDuration="1.69901792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:10.697266516 +0000 UTC m=+1200.886804193" watchObservedRunningTime="2026-10-30 22:03:10.69901792 +0000 UTC m=+1200.888555597" Oct 30 22:03:10.756524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.756446 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:10.802893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.802858 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:10.872662 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.872632 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw"] Oct 30 22:03:10.876760 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.876740 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:10.937704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:10.937671 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw"] Oct 30 22:03:11.003593 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.003545 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.012242 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.011428 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.012242 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.011597 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.028495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.015414 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvck\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.028495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.015485 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.028495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.015512 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.028495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.015557 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.028495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.015658 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.028495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.015765 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.118329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.117513 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvck\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.118329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.117583 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.118329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.117609 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.118329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.117641 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.118329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.117720 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.118329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.117771 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.126371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.125824 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.126371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.126093 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.126371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.126167 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.126371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.126223 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.126833 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.126790 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.127527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.127500 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.127656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.127503 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.127964 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.127943 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.129999 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.129981 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.130434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.130417 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.195225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.195144 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.200252 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.200185 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvck\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.526337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.506138 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:11.757342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.757057 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:11.757342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.757118 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:11.819393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.808233 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:11.819393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:11.808327 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:12.037465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.037433 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw"] Oct 30 22:03:12.042876 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.042836 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:12.042977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.042909 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:12.042977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.042939 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:12.573990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.573953 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" event={"ID":"9eca57e9-2749-4ccb-9266-5261b4772220","Type":"ContainerStarted","Data":"0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1"} Oct 30 22:03:12.574560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.574020 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" event={"ID":"9eca57e9-2749-4ccb-9266-5261b4772220","Type":"ContainerStarted","Data":"f0539634dc625dfa7b1be5e35ebb84af2e6781530f4448c843999c763eb96d71"} Oct 30 22:03:12.698328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.696814 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podStartSLOduration=2.696794731 podStartE2EDuration="2.696794731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:12.693847837 +0000 UTC m=+1202.883385513" watchObservedRunningTime="2026-10-30 22:03:12.696794731 +0000 UTC m=+1202.886332436" Oct 30 22:03:12.756283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.756241 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:12.756496 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.756318 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:12.836423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.816626 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:12.836423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:12.816684 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:13.526330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.519106 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:13.526330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.519235 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:13.526330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.519279 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:13.756624 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.756589 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:13.757054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.757031 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:13.804392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.803687 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:13.804392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:13.803747 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:14.512778 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.512737 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:14.513083 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.513055 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:14.758562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.757730 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:14.758562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.757830 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:14.809810 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.809714 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:14.810079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.810045 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:14.847933 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:14.847895 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:15.516794 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.516760 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:15.517069 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.517042 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:15.756357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.742426 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:15.757774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.757718 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:15.757905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.757818 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:15.817376 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.817263 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:15.817918 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.817882 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:15.939466 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:15.939417 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.191487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.191439 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:03:16.299334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.297414 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.435342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.433402 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:03:16.468476 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.465818 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.524331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.523766 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.524331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.523839 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:16.704583 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.704521 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.733597 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.733481 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.745430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.745383 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.746110 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.746087 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:03:16.788028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.787974 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.789638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.789590 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:16.832802 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.832709 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.833411 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:16.833366 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:17.120334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.095454 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:03:17.241332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.125032 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:03:17.274600 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.134536 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.274872 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.192188 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.292423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.292369 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:03:17.339286 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.338063 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:03:17.371589 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.371485 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:03:17.450330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.436964 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.450330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.447553 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:03:17.478905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.466741 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.478905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.467105 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:03:17.557804 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.557768 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.558067 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.558042 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:17.707623 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.707529 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.757572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.742445 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:03:17.757572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.744266 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.757572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.752748 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:03:17.773374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.758382 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:03:17.773374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.764405 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.773374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.764457 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:17.774258 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.774224 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.804190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.804126 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.804418 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:17.804224 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:18.202329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.199808 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:03:18.217952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.216397 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:03:18.285337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.280644 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 22:03:18.285337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.280913 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:03:18.285337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.281062 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:03:18.285337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.281266 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:03:18.451960 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.448413 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:03:18.451960 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.449552 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:03:18.534396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.524415 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.534396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.524481 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:18.549341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.489292 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.784329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.774712 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.784329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.774773 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 22:03:18.784730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.784656 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.933458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.784883 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:03:18.933458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.878802 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:03:18.933458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.885752 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.933458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:18.929774 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 22:03:19.019775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.019729 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.167593 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.167562 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:03:19.203429 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.203387 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.288847 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.288808 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.289424 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.289091 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.361394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.361360 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:03:19.451480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.451372 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.451979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.451952 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:03:19.562991 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.562944 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.571633 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.571601 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:19.571874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.571851 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:19.662190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.598652 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:03:19.749575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.749465 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:03:19.761727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.759598 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:03:19.761727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.760558 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:03:19.846961 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.846935 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:03:19.869818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.869790 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:03:19.871497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:19.871472 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:03:20.408331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.396180 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:20.423582 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.423553 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:20.423833 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.423819 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:03:20.512329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.511602 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:20.512329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.511662 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:20.809383 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.808997 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:20.809383 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:20.809051 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:21.524331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:21.523874 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:03:21.524331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:21.523949 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 22:03:21.863439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:21.863409 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.864459 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:21.864430 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:21.864566 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:21.864488 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:22.100060 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:22.100027 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:22.100662 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:22.100636 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:22.101489 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:22.101458 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:03:22.515518 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:22.515490 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:22.515928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:22.515911 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:22.516704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:22.516685 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:03:26.732391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:26.732280 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.704655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:27.704620 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:31.559838 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:31.559773 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.289369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:36.289335 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.032161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.032123 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7"] Oct 30 22:03:40.036416 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.036398 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.138633 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.138595 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.138812 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.138646 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.138812 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.138681 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pt68\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.138812 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.138715 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.138981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.138841 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.138981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.138956 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.139071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.139010 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.139071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.139050 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.139210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.139101 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.139261 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.139216 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7"] Oct 30 22:03:40.239980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.239942 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240030 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240063 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240095 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240126 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240189 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240213 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240247 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pt68\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240476 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240519 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240529 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240813 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240638 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.240987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240962 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.241045 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.240999 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.242617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.242591 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.242919 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.242902 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.297864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.297759 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pt68\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.298020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.297863 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.351098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.351067 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:40.376408 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:03:40.376372 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:03:40.384565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.384538 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv"] Oct 30 22:03:40.388953 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.388932 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443034 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443197 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443077 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443241 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443208 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443251 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443344 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443544 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443521 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443616 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443563 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443658 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443645 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.443698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.443679 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwz4l\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.451186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.449855 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv"] Oct 30 22:03:40.544771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.544734 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.544771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.544780 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.544805 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.544836 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwz4l\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.544896 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.544926 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545027 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545290 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545074 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545290 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545102 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545449 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545487 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545745 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.545900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.545884 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.546488 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.546441 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.548179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.547982 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.548179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.548140 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.603935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.603910 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwz4l\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.604784 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.604767 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.706191 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.706165 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:40.794469 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.789002 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7"] Oct 30 22:03:40.795390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.795138 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:40.795390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.795221 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:40.795390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:40.795265 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.101117 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.101089 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv"] Oct 30 22:03:41.103427 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:03:41.103395 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ae515e_2442_4885_b4d9_d1d54a2d8eda.slice/crio-8f4654edeb7ecd89a521ba0b4fec3236b1eff628aa622cb22358afc50aba87ef WatchSource:0}: Error finding container 8f4654edeb7ecd89a521ba0b4fec3236b1eff628aa622cb22358afc50aba87ef: Status 404 returned error can't find the container with id 8f4654edeb7ecd89a521ba0b4fec3236b1eff628aa622cb22358afc50aba87ef Oct 30 22:03:41.106162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.106128 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.106238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.106197 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.106238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.106227 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.157611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.157575 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" event={"ID":"03ecca1a-530c-4f48-9e42-a7d7408ba81f","Type":"ContainerStarted","Data":"619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6"} Oct 30 22:03:41.157827 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.157647 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" event={"ID":"03ecca1a-530c-4f48-9e42-a7d7408ba81f","Type":"ContainerStarted","Data":"7a6f57e737c802695c76cac9942df66dbe8be7cdecd2ac62e3cb95c7f54eb37c"} Oct 30 22:03:41.159169 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.159139 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" event={"ID":"03ae515e-2442-4885-b4d9-d1d54a2d8eda","Type":"ContainerStarted","Data":"8f4654edeb7ecd89a521ba0b4fec3236b1eff628aa622cb22358afc50aba87ef"} Oct 30 22:03:41.278865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.278809 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podStartSLOduration=2.278795137 podStartE2EDuration="2.278795137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:41.274711419 +0000 UTC m=+1231.464249201" watchObservedRunningTime="2026-10-30 22:03:41.278795137 +0000 UTC m=+1231.468332811" Oct 30 22:03:41.290837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.290787 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:41.351863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.351790 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:41.353381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.353293 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:41.353514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:41.353407 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:42.167237 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.167198 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" event={"ID":"03ae515e-2442-4885-b4d9-d1d54a2d8eda","Type":"ContainerStarted","Data":"e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543"} Oct 30 22:03:42.341651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.341597 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podStartSLOduration=2.34158106 podStartE2EDuration="2.34158106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:42.338764692 +0000 UTC m=+1232.528302381" watchObservedRunningTime="2026-10-30 22:03:42.34158106 +0000 UTC m=+1232.531118736" Oct 30 22:03:42.351856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.351800 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:42.351856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.351845 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:42.716332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.710354 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:42.716332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.710491 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:03:42.716332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.710537 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:03:42.806332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.806208 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg"] Oct 30 22:03:42.829921 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.829896 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.854551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.854526 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg"] Oct 30 22:03:42.886328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.884358 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.918627 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5jfgf\"" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.918723 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.918751 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924245 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924462 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924523 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88dbae2-8521-4946-a12b-7d44e60b04f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924670 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924708 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924738 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:42.928771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:42.924778 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc2p8\" (UniqueName: \"kubernetes.io/projected/b88dbae2-8521-4946-a12b-7d44e60b04f1-kube-api-access-dc2p8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.025775 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.025949 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.025998 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88dbae2-8521-4946-a12b-7d44e60b04f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.026212 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.026662 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026802 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.026697 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026802 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.026736 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc2p8\" (UniqueName: \"kubernetes.io/projected/b88dbae2-8521-4946-a12b-7d44e60b04f1-kube-api-access-dc2p8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.026943 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.026915 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.029419 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.029323 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.029825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.029665 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.029825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.029791 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.030663 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.030645 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b88dbae2-8521-4946-a12b-7d44e60b04f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.094339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.092900 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b88dbae2-8521-4946-a12b-7d44e60b04f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.102339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.101958 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc2p8\" (UniqueName: \"kubernetes.io/projected/b88dbae2-8521-4946-a12b-7d44e60b04f1-kube-api-access-dc2p8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r8zrg\" (UID: \"b88dbae2-8521-4946-a12b-7d44e60b04f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.143871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.143849 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:43.352442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.352407 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:43.352813 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.352467 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:43.628670 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.628428 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg"] Oct 30 22:03:43.629880 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:03:43.629851 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88dbae2_8521_4946_a12b_7d44e60b04f1.slice/crio-da75905cb746b61bca239776e65a99b910b33655ae20a52a51515433994964e3 WatchSource:0}: Error finding container da75905cb746b61bca239776e65a99b910b33655ae20a52a51515433994964e3: Status 404 returned error can't find the container with id da75905cb746b61bca239776e65a99b910b33655ae20a52a51515433994964e3 Oct 30 22:03:43.728334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.721828 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:03:43.728334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:43.721918 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:03:44.231026 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:44.230980 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" event={"ID":"b88dbae2-8521-4946-a12b-7d44e60b04f1","Type":"ContainerStarted","Data":"da75905cb746b61bca239776e65a99b910b33655ae20a52a51515433994964e3"} Oct 30 22:03:44.352211 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:44.352165 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:44.352391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:44.352225 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:44.722474 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:44.722432 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:03:44.722988 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:44.722497 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:03:45.351639 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:45.351604 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:45.351823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:45.351668 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:45.710399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:45.708746 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:03:45.710399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:45.708820 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:03:46.326329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:46.318628 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:46.354643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:46.354603 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:46.354866 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:46.354840 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:46.708806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:46.708712 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:03:46.709076 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:46.709047 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:03:47.352079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:47.352040 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:03:47.352591 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:47.352116 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:03:47.707482 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:47.707390 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:03:47.707482 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:47.707462 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:03:48.682280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:48.682241 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:48.683979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:48.683751 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:48.684582 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:48.684559 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:03:48.743396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:48.743362 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:48.743511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:48.743435 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:49.013671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.013584 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:49.015051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.015025 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:49.016752 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.016728 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:03:49.421324 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.420476 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" event={"ID":"b88dbae2-8521-4946-a12b-7d44e60b04f1","Type":"ContainerStarted","Data":"e356fb6d2364bcfb5b4b2ca39bb07ce9ab0492cf3bac1fa9e7d759f04c212b1e"} Oct 30 22:03:49.421324 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.420573 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:49.421324 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.420726 5508 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r8zrg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:49.421324 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:49.420783 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" podUID="b88dbae2-8521-4946-a12b-7d44e60b04f1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:50.496334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:50.493514 5508 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r8zrg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:03:50.496334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:50.493574 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" podUID="b88dbae2-8521-4946-a12b-7d44e60b04f1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:03:51.467672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:51.467636 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:03:51.467957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:51.467932 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:51.487491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:51.484403 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:52.288165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:52.288128 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:53.474263 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:53.474235 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" Oct 30 22:03:53.703996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:53.703942 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r8zrg" podStartSLOduration=6.593943474 podStartE2EDuration="11.703927059s" totalImagesPullingTime="5.109983585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:49.577856339 +0000 UTC m=+1239.767394043" watchObservedRunningTime="2026-10-30 22:03:53.703927059 +0000 UTC m=+1243.893464734" Oct 30 22:03:57.286268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:03:57.286240 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:02.087664 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:02.087614 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:05.183212 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:05.183181 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j"] Oct 30 22:04:05.183689 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:05.183397 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" containerID="cri-o://8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5" gracePeriod=30 Oct 30 22:04:05.224755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:05.224727 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l"] Oct 30 22:04:05.224979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:05.224937 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" containerID="cri-o://1dec334063968c3c83adeb5b938cced18679bfdbe9dc9f7703701d3853c3cc56" gracePeriod=30 Oct 30 22:04:06.952823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:06.952780 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:09.322341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.322284 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5"] Oct 30 22:04:09.326821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.326801 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.385889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.385851 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.385903 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.385930 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.385982 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.386058 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.386113 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5"] Oct 30 22:04:09.386511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.386339 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.386389 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.386450 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.386511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.386493 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkhn\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487465 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487503 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487542 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487581 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkhn\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487813 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487708 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487813 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487743 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487813 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487766 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487814 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.487972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.487886 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.488148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.488118 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.488252 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.488156 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.488360 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.488340 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.488418 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.488354 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.488582 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.488557 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.490236 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.490205 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.490518 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.490490 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.545502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.545481 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.545639 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.545622 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkhn\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:09.640998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:09.640891 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:10.076756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.076726 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5"] Oct 30 22:04:10.077831 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:04:10.077800 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda608a1_3b58_445c_b3c7_9e75d4044544.slice/crio-95e30111338e5f3f4080085356f97914979a11c96026753433b36c8e385e3192 WatchSource:0}: Error finding container 95e30111338e5f3f4080085356f97914979a11c96026753433b36c8e385e3192: Status 404 returned error can't find the container with id 95e30111338e5f3f4080085356f97914979a11c96026753433b36c8e385e3192 Oct 30 22:04:10.080371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.080341 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:10.080433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.080400 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:10.080470 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.080432 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:10.386423 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:04:10.386376 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:04:10.635552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.635520 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 22:04:10.668956 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.668917 5508 generic.go:356] "Generic (PLEG): container finished" podID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerID="1dec334063968c3c83adeb5b938cced18679bfdbe9dc9f7703701d3853c3cc56" exitCode=0 Oct 30 22:04:10.669093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.669037 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" event={"ID":"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8","Type":"ContainerDied","Data":"1dec334063968c3c83adeb5b938cced18679bfdbe9dc9f7703701d3853c3cc56"} Oct 30 22:04:10.672501 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.671998 5508 generic.go:356] "Generic (PLEG): container finished" podID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerID="8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5" exitCode=0 Oct 30 22:04:10.672501 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.672146 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" event={"ID":"fd8f2321-b027-4336-960d-03e9b29ff77f","Type":"ContainerDied","Data":"8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5"} Oct 30 22:04:10.672501 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.672191 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" event={"ID":"fd8f2321-b027-4336-960d-03e9b29ff77f","Type":"ContainerDied","Data":"e1a9322c3a767827ba945320a0639899fac02b63552d796d3ecd8fa2fe127ddb"} Oct 30 22:04:10.672501 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.672221 5508 scope.go:122] "RemoveContainer" containerID="8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5" Oct 30 22:04:10.672501 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.672390 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j" Oct 30 22:04:10.676954 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.676915 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" event={"ID":"bda608a1-3b58-445c-b3c7-9e75d4044544","Type":"ContainerStarted","Data":"dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e"} Oct 30 22:04:10.677077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.676989 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" event={"ID":"bda608a1-3b58-445c-b3c7-9e75d4044544","Type":"ContainerStarted","Data":"95e30111338e5f3f4080085356f97914979a11c96026753433b36c8e385e3192"} Oct 30 22:04:10.686798 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.686774 5508 scope.go:122] "RemoveContainer" containerID="8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5" Oct 30 22:04:10.687127 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:04:10.687097 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5\": container with ID starting with 8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5 not found: ID does not exist" containerID="8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5" Oct 30 22:04:10.687224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.687140 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5"} err="failed to get container status \"8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5\": rpc error: code = NotFound desc = could not find container \"8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5\": container with ID starting with 8a9f1a8524cd49843f2c11200633b354cf5657808b10caadf4d7205357763ba5 not found: ID does not exist" Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.741608 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.741707 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.741777 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.741887 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.741916 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.741982 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.742080 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.742115 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.742152 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.742203 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket\") pod \"fd8f2321-b027-4336-960d-03e9b29ff77f\" (UID: \"fd8f2321-b027-4336-960d-03e9b29ff77f\") " Oct 30 22:04:10.742958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.742915 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.743263 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.743239 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.743497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.743349 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:10.743497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.743419 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.743633 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.743593 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.750601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.750519 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:10.750601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.750552 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "kube-api-access-2t9sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.751524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.751077 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.761690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.761665 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 22:04:10.772107 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.772080 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token" pod "fd8f2321-b027-4336-960d-03e9b29ff77f" (UID: "fd8f2321-b027-4336-960d-03e9b29ff77f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843439 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843495 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843596 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843707 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843824 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843872 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843927 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.843961 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.844085 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy\") pod \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\" (UID: \"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8\") " Oct 30 22:04:10.844998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.844971 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845004 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f2321-b027-4336-960d-03e9b29ff77f-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845021 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845036 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845049 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845063 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845300 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845077 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9sm\" (UniqueName: \"kubernetes.io/projected/fd8f2321-b027-4336-960d-03e9b29ff77f-kube-api-access-2t9sm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845300 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845093 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f2321-b027-4336-960d-03e9b29ff77f-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.845740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.845716 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:10.850058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.848565 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.850058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.848787 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.850058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.848977 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "kube-api-access-n2nz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.850058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.849423 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.850058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.849587 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.850428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.850405 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:10.852506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.852476 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:10.855093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.853621 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy" pod "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" (UID: "0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:10.967512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967466 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nz2\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-kube-api-access-n2nz2\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967507 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967523 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967537 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967552 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967565 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967578 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967591 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:10.967768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:10.967606 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:11.345338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.336125 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podStartSLOduration=2.336102641 podStartE2EDuration="2.336102641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:11.194831993 +0000 UTC m=+1261.384369681" watchObservedRunningTime="2026-10-30 22:04:11.336102641 +0000 UTC m=+1261.525640317" Oct 30 22:04:11.673333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.669538 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:11.679336 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.674788 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j"] Oct 30 22:04:11.688360 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.685634 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:11.688360 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.685691 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:11.846181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.846101 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dgd2j"] Oct 30 22:04:11.856394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.849477 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" event={"ID":"0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8","Type":"ContainerDied","Data":"c3e8c5d257a549fc6a3454ed9f62ff7990ac86085b37d9db4855a3b96461a959"} Oct 30 22:04:11.856394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.849545 5508 scope.go:122] "RemoveContainer" containerID="1dec334063968c3c83adeb5b938cced18679bfdbe9dc9f7703701d3853c3cc56" Oct 30 22:04:11.856394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:11.849672 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l" Oct 30 22:04:12.108794 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.108691 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" path="/var/lib/kubelet/pods/fd8f2321-b027-4336-960d-03e9b29ff77f/volumes" Oct 30 22:04:12.141350 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:04:12.129647 5508 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b83c8_8b9c_4039_bb65_c0c180cf2dc8.slice\": RecentStats: unable to find data in memory cache]" Oct 30 22:04:12.168333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.167704 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l"] Oct 30 22:04:12.201332 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:04:12.174131 5508 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b83c8_8b9c_4039_bb65_c0c180cf2dc8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b83c8_8b9c_4039_bb65_c0c180cf2dc8.slice/crio-c3e8c5d257a549fc6a3454ed9f62ff7990ac86085b37d9db4855a3b96461a959\": RecentStats: unable to find data in memory cache]" Oct 30 22:04:12.201332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.198883 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c4k6l"] Oct 30 22:04:12.312343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.309490 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.439442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.430125 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.492395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.483015 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.560434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.560388 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.575645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.575598 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.659332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.646846 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:12.659332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.646911 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:12.761403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.759457 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.794342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.765843 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.882327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.881405 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.955331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:12.954912 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.335342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.313871 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.335342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.315614 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:04:13.519332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.473913 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:04:13.599386 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.525252 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:04:13.627783 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.564788 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.628184 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.578889 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:04:13.718463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.718427 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:04:13.718744 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.718727 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:04:13.808859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.579280 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.808859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.672209 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:13.808859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.744601 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:13.808859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.744685 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:04:13.808859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.776391 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.808859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.781408 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:04:13.823338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.815997 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:04:13.879396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.826543 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:04:13.879396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.847637 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:13.959675 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.902376 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:13.959675 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:13.931517 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:04:14.028331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.025521 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:04:14.612335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.610745 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:14.646347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.629543 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.646347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.630911 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.646347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.631105 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:04:14.789330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.788388 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.874773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.873888 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:04:15.014328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:14.965549 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:15.211793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.061600 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:04:15.223010 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.150906 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.223278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.223246 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:15.223491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.175482 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:04:15.536381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.536184 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" path="/var/lib/kubelet/pods/0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8/volumes" Oct 30 22:04:15.574763 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:04:15.574727 5508 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.616s" Oct 30 22:04:15.575204 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.575181 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:04:15.575528 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.575498 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:04:15.575724 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.575711 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:04:15.575916 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.575904 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:04:15.630704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.630647 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.644694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.644661 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.656321 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.646031 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:15.669106 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.669047 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:04:15.670686 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.670192 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:04:15.670686 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.670249 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:04:15.670686 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.670283 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:04:15.672134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.672110 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:04:15.715888 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:15.715865 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:04:16.641910 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:16.641873 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.642366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:16.641924 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:17.642186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:17.642150 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:17.642599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:17.642205 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:18.642424 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:18.642331 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.642424 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:18.642411 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:19.641503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:19.641460 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.641672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:19.641530 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:04:20.617960 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:20.617928 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:20.645912 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:20.645886 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:20.646380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:20.646349 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:20.647059 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:20.647040 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:04:25.499090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:25.499053 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:28.050493 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:28.050459 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:30.308031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:30.307998 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:35.216984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:35.216939 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:40.080514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:40.080481 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:40.373877 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:04:40.373791 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:04:41.298171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.298132 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg"] Oct 30 22:04:41.298973 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.298950 5508 cpu_manager.go:414] "containerMap: removing container" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" Oct 30 22:04:41.299072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.298975 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" Oct 30 22:04:41.299072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.299041 5508 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" Oct 30 22:04:41.299072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.299050 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" Oct 30 22:04:41.299225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.299207 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f2321-b027-4336-960d-03e9b29ff77f" containerName="istio-proxy" Oct 30 22:04:41.299275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.299231 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6b83c8-8b9c-4039-bb65-c0c180cf2dc8" containerName="istio-proxy" Oct 30 22:04:41.302874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.302852 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.370589 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370561 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.370740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370605 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.370788 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370727 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.370826 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370786 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfcw\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.370903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370886 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.370961 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370939 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.371014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.370968 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.371117 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.371099 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.371238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.371220 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.388724 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.388687 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg"] Oct 30 22:04:41.474161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474125 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474179 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474236 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474281 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfcw\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474416 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474466 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474492 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474624 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474796 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.474977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474953 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.475042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.474960 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.475218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.475173 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.475339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.475278 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.475739 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.475716 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.477103 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.477082 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.477435 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.477414 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.534761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.534736 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.534965 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.534943 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfcw\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.617783 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.617761 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:41.637006 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.636980 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw"] Oct 30 22:04:41.642328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.642291 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.718230 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.718193 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw"] Oct 30 22:04:41.780062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780024 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780075 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780199 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780228 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780394 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780484 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcvk\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780537 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780662 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780647 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.780696 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.780682 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.881940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.881850 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.881940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.881908 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882011 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882038 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882291 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882188 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882291 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882233 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882333 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882412 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcvk\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882421 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882451 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882459 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.882928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882900 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.883027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.882927 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.883368 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.883350 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.885256 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.885238 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.885559 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.885541 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.938770 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.938748 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.938906 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.938889 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcvk\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:41.962756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:41.962727 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:42.061533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.061501 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg"] Oct 30 22:04:42.063032 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:04:42.063006 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6976822_f864_4d39_8902_576c8e167e82.slice/crio-50eb24ea420857be7913d139834f78a2602590715a22da8cd222e20593e69a95 WatchSource:0}: Error finding container 50eb24ea420857be7913d139834f78a2602590715a22da8cd222e20593e69a95: Status 404 returned error can't find the container with id 50eb24ea420857be7913d139834f78a2602590715a22da8cd222e20593e69a95 Oct 30 22:04:42.067784 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.067740 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.067901 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.067864 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.067968 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.067911 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.350151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.350125 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw"] Oct 30 22:04:42.351735 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:04:42.351706 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66b2e30_b7ec_4006_a6fe_8f5b288d9c5f.slice/crio-b5ebde2931f9e1acbd532849b72c3a54333838473dd22c352c1b5c643162f513 WatchSource:0}: Error finding container b5ebde2931f9e1acbd532849b72c3a54333838473dd22c352c1b5c643162f513: Status 404 returned error can't find the container with id b5ebde2931f9e1acbd532849b72c3a54333838473dd22c352c1b5c643162f513 Oct 30 22:04:42.354478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.354441 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.354571 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.354522 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.354639 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.354567 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.636499 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.636464 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" event={"ID":"d6976822-f864-4d39-8902-576c8e167e82","Type":"ContainerStarted","Data":"4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487"} Oct 30 22:04:42.636718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.636528 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" event={"ID":"d6976822-f864-4d39-8902-576c8e167e82","Type":"ContainerStarted","Data":"50eb24ea420857be7913d139834f78a2602590715a22da8cd222e20593e69a95"} Oct 30 22:04:42.638354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.638322 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" event={"ID":"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f","Type":"ContainerStarted","Data":"e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833"} Oct 30 22:04:42.638486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.638394 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" event={"ID":"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f","Type":"ContainerStarted","Data":"b5ebde2931f9e1acbd532849b72c3a54333838473dd22c352c1b5c643162f513"} Oct 30 22:04:42.756403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.756276 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podStartSLOduration=1.756263356 podStartE2EDuration="1.756263356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.753881622 +0000 UTC m=+1292.943419313" watchObservedRunningTime="2026-10-30 22:04:42.756263356 +0000 UTC m=+1292.945801110" Oct 30 22:04:42.873909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.873858 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podStartSLOduration=1.8738426430000001 podStartE2EDuration="1.873842643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.871905206 +0000 UTC m=+1293.061442883" watchObservedRunningTime="2026-10-30 22:04:42.873842643 +0000 UTC m=+1293.063380319" Oct 30 22:04:42.963228 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:42.963200 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:43.621033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:43.620566 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:43.621033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:43.620716 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:43.621033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:43.620767 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:43.964660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:43.964585 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:43.964660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:43.964634 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:44.622752 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:44.622717 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:44.623375 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:44.623341 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:44.963755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:44.963673 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:44.963898 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:44.963774 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:45.509325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:45.508471 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:45.620768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:45.620730 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:45.621066 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:45.621030 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:45.963604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:45.963565 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:45.963999 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:45.963622 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:46.623332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:46.623199 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.623332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:46.623259 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:46.963643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:46.963563 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.963643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:46.963615 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:47.626709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:47.620765 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:47.626709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:47.620829 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:47.984835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:47.984285 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:47.984835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:47.984384 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:48.625762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.625727 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:48.651438 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.651397 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:48.651684 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.650758 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.651898 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.650452 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:48.685354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.672393 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:04:48.732820 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.731507 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.755458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.755414 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.964327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.964218 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:48.964590 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:48.964561 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:49.630331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:49.629871 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:49.630331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:49.629953 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:50.147421 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.115473 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.147421 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.115552 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:50.413386 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.398408 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.660343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.652498 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.664474 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.660929 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 22:04:50.676745 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.659968 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:04:50.690329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.659929 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.736283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.694744 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.736997 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.704549 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:04:50.799030 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.705537 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:50.815736 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.733924 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.815736 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:50.737723 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.010396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.009918 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:04:51.010396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.009975 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 22:04:51.252161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.252117 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.267751 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.254042 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Oct 30 22:04:51.424359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.417691 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.450335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.436294 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:04:51.693688 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.693641 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.694670 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.694644 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:04:51.708331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.696029 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:51.740784 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.738565 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.741370 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.740418 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:51.741525 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.741502 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.741638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.741060 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:04:51.742250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.742233 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:04:51.808401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.808355 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.822364 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.808386 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.823453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.808420 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:04:51.979390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.808451 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.982157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:51.822634 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:04:52.010023 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.009996 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:04:52.015442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.015419 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:04:52.025434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.025409 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:04:52.271388 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.270932 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:04:52.421369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.421297 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:04:52.505332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.495953 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:04:52.723331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.698567 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.741250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.741208 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.741878 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.741840 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.755630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.755597 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:52.755876 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.755850 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.824332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.823381 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:52.852621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.832395 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:52.949331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.929688 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:04:52.989672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:52.989579 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.045461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.045422 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:53.045727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.045697 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.325242 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.324569 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:04:53.444354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.440799 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.698705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.698608 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:53.698948 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.698921 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.734447 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.734401 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:04:53.776012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.775963 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:04:53.776354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.776296 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:53.825343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.822855 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:53.825343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.822925 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.825343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.824082 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.825765 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.825702 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" podUID="7355813a-a26d-4321-918a-c206560e85a3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.848387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.848352 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:53.950418 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.939561 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:53.965818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.952393 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:04:54.014332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:53.996511 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.218396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.195939 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:54.218396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.196024 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:04:54.340868 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.340824 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.475936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.475552 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.749650 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.749563 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:04:54.751635 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.751602 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.781339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.777041 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.781339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.777536 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.807690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.802854 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:04:54.810337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.809598 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:04:54.830475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.830424 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:54.830673 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.830652 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.854075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.853392 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.996330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:54.989529 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:55.061370 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.061256 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:04:55.061762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.061746 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:04:55.069400 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.069373 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:04:55.069943 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.069919 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:04:55.073249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.073200 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:04:55.155556 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.155511 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:04:55.842546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.842504 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:55.842896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.842567 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:55.845048 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.845017 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:55.845447 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.845429 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:55.846249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:55.846232 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:04:56.381847 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.381818 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p"] Oct 30 22:04:56.385842 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.385822 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.445527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.445500 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p"] Oct 30 22:04:56.454053 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454026 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454089 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454180 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454322 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454391 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657gr\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454436 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454484 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454523 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.454647 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.454553 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.555778 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.555748 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.555946 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.555813 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.555946 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.555849 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.555946 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.555923 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.555953 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657gr\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.555996 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556083 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556106 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556263 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556411 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556433 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556484 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556570 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.556857 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.556827 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.558389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.558369 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.558685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.558664 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.615713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.615683 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657gr\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.615819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.615711 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:56.700539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:56.700467 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:57.138253 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:57.138226 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p"] Oct 30 22:04:57.140961 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:04:57.140936 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53062e4_bdde_401c_a512_465b522fcd64.slice/crio-82d58aa29b3e92711bc71976ae3b1d46bdfc5820fceae23b158f7127e7844e88 WatchSource:0}: Error finding container 82d58aa29b3e92711bc71976ae3b1d46bdfc5820fceae23b158f7127e7844e88: Status 404 returned error can't find the container with id 82d58aa29b3e92711bc71976ae3b1d46bdfc5820fceae23b158f7127e7844e88 Oct 30 22:04:57.144207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:57.144170 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:57.144356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:57.144239 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:57.144356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:57.144272 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:57.166973 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:57.166936 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" event={"ID":"a53062e4-bdde-401c-a512-465b522fcd64","Type":"ContainerStarted","Data":"82d58aa29b3e92711bc71976ae3b1d46bdfc5820fceae23b158f7127e7844e88"} Oct 30 22:04:57.810553 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:57.810496 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:58.205397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:58.204351 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" event={"ID":"a53062e4-bdde-401c-a512-465b522fcd64","Type":"ContainerStarted","Data":"92c6e0b1a6a3e8ccbfe1db4d0f07da6f8072de0c21646f349c0c62560f699f33"} Oct 30 22:04:58.703718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:58.703679 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:04:58.705018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:58.704955 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:04:58.705018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:58.705020 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:04:59.707641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:59.706071 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:04:59.707641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:04:59.706133 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:05:00.707598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:00.705792 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:05:00.707598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:00.705856 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:05:01.700957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:01.700912 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:05:01.701263 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:01.701231 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:05:02.701111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:02.701073 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:05:02.701624 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:02.701138 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:05:03.273634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:03.273604 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:03.701266 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:03.701229 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:05:03.701798 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:03.701281 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:05:04.705029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:04.705000 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:05:04.705412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:04.705264 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:05:04.706124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:04.706098 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:05:04.858527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:04.858476 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podStartSLOduration=8.858462475 podStartE2EDuration="8.858462475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:58.41866729 +0000 UTC m=+1308.608204984" watchObservedRunningTime="2026-10-30 22:05:04.858462475 +0000 UTC m=+1315.048000205" Oct 30 22:05:08.304895 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:08.304767 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:09.582382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:09.582293 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:10.380237 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:05:10.380158 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:05:10.528911 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:10.528720 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:10.760759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:10.760657 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:10.976271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:10.976232 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:11.568949 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:11.568912 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:13.963621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:13.963576 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.170065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:14.170015 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.630183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:14.630138 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.819820 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:14.819787 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.992534 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:15.992498 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:16.172200 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:16.172162 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:17.326675 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:17.326585 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:17.518884 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:17.518846 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:18.749854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:18.749813 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:18.915955 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:18.915924 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.092045 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:19.092015 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.335877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:19.335847 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.530445 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:19.530356 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.749327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:19.745790 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:19.944874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:19.944843 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:20.131330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:20.129490 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.615529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:22.615494 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.809923 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:22.809892 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:23.986478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:23.986443 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:24.169260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:24.169225 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:24.378645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:24.378614 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:24.937111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:24.937079 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk"] Oct 30 22:05:24.942950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:24.942929 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.050611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.050576 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.050992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.050643 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.050992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.050894 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.050992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.050945 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.051110 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.051057 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.051221 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.051204 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.051283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.051239 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.051487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.051462 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.051644 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.051511 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kc2\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.144947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.144912 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk"] Oct 30 22:05:25.153265 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153234 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.153529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153504 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.153638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153615 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.153732 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153642 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kc2\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.153732 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153681 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.153837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153739 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.153914 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153899 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.154024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153938 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.154024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.153980 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.154464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.154442 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.155100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.155074 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.155362 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.155332 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.156193 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.156164 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.156472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.156450 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.159350 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.159279 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.169962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.169930 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.213142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.213082 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.213255 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.213178 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kc2\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.258325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.258289 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:25.533190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.533106 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:25.692947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.692916 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk"] Oct 30 22:05:25.693590 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:05:25.693564 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb72fcb6_dc6a_4fcd_ac79_555c185eaa8d.slice/crio-5e74161a4c942e26975cfc848542a0d26c7c3bd12ea0b7aeb01195e1915553b2 WatchSource:0}: Error finding container 5e74161a4c942e26975cfc848542a0d26c7c3bd12ea0b7aeb01195e1915553b2: Status 404 returned error can't find the container with id 5e74161a4c942e26975cfc848542a0d26c7c3bd12ea0b7aeb01195e1915553b2 Oct 30 22:05:25.696264 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.696227 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:25.696390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.696297 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:25.696390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:25.696374 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:26.420788 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:26.420748 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" event={"ID":"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d","Type":"ContainerStarted","Data":"647368fdf6e1f8b25ffa360ad743b660ffa8c1fea82afbbd2bf2549f9638b6b1"} Oct 30 22:05:26.421174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:26.420807 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" event={"ID":"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d","Type":"ContainerStarted","Data":"5e74161a4c942e26975cfc848542a0d26c7c3bd12ea0b7aeb01195e1915553b2"} Oct 30 22:05:26.602700 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:26.602643 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podStartSLOduration=2.602626851 podStartE2EDuration="2.602626851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:26.597606567 +0000 UTC m=+1336.787144249" watchObservedRunningTime="2026-10-30 22:05:26.602626851 +0000 UTC m=+1336.792164525" Oct 30 22:05:27.259172 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:27.259135 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:27.260635 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:27.260604 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.260769 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:27.260660 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:27.442936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:27.442887 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.747427 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:27.747391 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:28.259229 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:28.259191 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.259458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:28.259241 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:28.518276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:28.518173 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.264439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:29.264400 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:29.264731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:29.264697 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:30.269332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:30.262415 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:30.269332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:30.262485 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:30.778478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:30.777957 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:30.908704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:30.908665 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.006330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.002757 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.058891 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.058517 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.181997 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.181660 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.400333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.347484 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.432861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.432828 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:31.433119 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.433095 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:31.492221 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.490977 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:05:31.586155 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.586100 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:31.586505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.586455 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.744573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.744451 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.749495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.749445 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.819330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.783124 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.836330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.825828 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:05:31.909443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.909394 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.911544 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:31.911519 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:05:32.008009 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.007917 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:05:32.035910 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.035871 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:32.059069 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.059011 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:32.363392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.281374 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:05:32.363392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.281439 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:05:32.363392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.314895 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:32.363392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.314952 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:32.423331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.420045 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:05:32.423331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.420655 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:05:32.755405 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.540447 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:05:32.756031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.756002 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:05:32.756998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.756970 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:05:32.757434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.757415 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:05:32.786383 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.786332 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:05:32.788668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.788633 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:05:32.837659 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.837611 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:32.913455 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:32.913404 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:05:33.022497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.003253 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:33.022497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.010045 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.375332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.349901 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:05:33.375841 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.375797 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.400492 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.422964 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.430971 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.541356 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.439772 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.541473 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.565332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.423039 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.744414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.722680 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:33.763464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.763412 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.787697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.787653 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.789270 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.766860 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.869486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:33.869436 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" podUID="7355813a-a26d-4321-918a-c206560e85a3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.020414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.020287 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:05:34.043118 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.020781 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.044008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.021117 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:34.045144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.037353 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:34.294684 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.294588 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:34.297399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.297367 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:34.301345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.296916 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:34.395346 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.385016 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.524333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.523154 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:05:34.549209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.546415 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.791912 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.791859 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.801443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.801344 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:05:34.820439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:34.819947 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:05:35.032334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.031820 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:35.032334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.031940 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.075391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.058487 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:35.075391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.058886 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.075391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.059160 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:05:35.175330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.171421 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2xtqf" podUID="7355813a-a26d-4321-918a-c206560e85a3" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.396839 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.396743 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:35.397173 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.397141 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:35.408258 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.404075 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.531046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.531003 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.560363 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.555525 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.607535 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.607411 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.852895 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.852836 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:05:35.858230 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.858179 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.859263 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.859226 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:05:35.975754 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.974173 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.995431 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.995386 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:35.995720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:35.995689 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.104394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.100667 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.108422 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.101987 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:36.113350 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.102039 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:05:36.127746 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.102062 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:05:36.133216 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.102087 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.319983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.319946 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:36.332338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.320280 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:36.451873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.442431 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.507490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.507451 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:36.511330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.507731 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.552328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.551883 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.600328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.583505 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:05:36.874338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.865960 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.874338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.867379 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:05:36.884346 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:36.881580 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.123713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.109105 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:37.151448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.151338 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.152282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.152250 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.314959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.314921 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:37.342808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.321370 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:37.487168 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.459324 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:05:37.557609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.557561 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.666493 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.666445 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.800025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.798830 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:05:37.871318 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.870519 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.872780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.871610 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.933533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:37.932420 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.054463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.054373 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:05:38.082330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.067614 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:05:38.082330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.076408 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 22:05:38.082330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.082163 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:05:38.128330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.122412 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:05:38.144356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.141678 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:05:38.157426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.157391 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.179396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.178715 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:05:38.187614 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.187588 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:05:38.189480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.189390 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:05:38.259129 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.259084 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 22:05:38.259340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.259134 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 22:05:38.433950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:38.433873 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:40.259373 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:40.259342 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:40.300482 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:40.300434 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:40.300660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:40.300486 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:40.379562 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:05:40.379528 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:05:40.408199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:40.408165 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:40.999560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:40.999527 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:05:41.101923 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:41.101890 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:42.434541 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:42.434501 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:42.591917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:42.591865 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:46.073028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:46.072982 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:50.871141 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:50.871063 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:51.145903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.145815 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg"] Oct 30 22:05:51.151298 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.151277 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194205 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194175 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194212 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194232 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194549 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194431 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194654 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194637 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmfw\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194689 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194805 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194835 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.194867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.194856 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.195147 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.195125 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k"] Oct 30 22:05:51.199225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.199209 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.250859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.250832 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg"] Oct 30 22:05:51.296045 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296014 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmfw\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296069 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296113 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead1be97-52c5-489a-b659-f9342332885a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296166 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296194 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296216 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296243 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296273 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296392 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead1be97-52c5-489a-b659-f9342332885a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296495 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296555 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296587 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296618 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead1be97-52c5-489a-b659-f9342332885a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296624 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296663 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65m4\" (UniqueName: \"kubernetes.io/projected/ead1be97-52c5-489a-b659-f9342332885a-kube-api-access-f65m4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296683 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296905 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.296959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296931 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.297671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.296977 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.297671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.297117 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.297671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.297173 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.297671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.297636 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.297873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.297750 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.301837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.301367 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.314437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.306658 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.318327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.316776 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k"] Oct 30 22:05:51.354942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.354915 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmfw\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.356585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.356560 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.398609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398548 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.398609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398594 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.398755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398686 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead1be97-52c5-489a-b659-f9342332885a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.398808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398728 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.398877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398858 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.398952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398939 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead1be97-52c5-489a-b659-f9342332885a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.398994 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead1be97-52c5-489a-b659-f9342332885a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399012 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399042 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65m4\" (UniqueName: \"kubernetes.io/projected/ead1be97-52c5-489a-b659-f9342332885a-kube-api-access-f65m4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399052 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399213 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399111 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399243 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399496 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.399588 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.399500 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ead1be97-52c5-489a-b659-f9342332885a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.401046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.401023 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ead1be97-52c5-489a-b659-f9342332885a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.401563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.401539 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ead1be97-52c5-489a-b659-f9342332885a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.455531 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.455508 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ead1be97-52c5-489a-b659-f9342332885a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.455915 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.455897 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65m4\" (UniqueName: \"kubernetes.io/projected/ead1be97-52c5-489a-b659-f9342332885a-kube-api-access-f65m4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k\" (UID: \"ead1be97-52c5-489a-b659-f9342332885a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.466766 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.466748 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:51.512622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.512420 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:51.897032 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.897000 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg"] Oct 30 22:05:51.898035 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:05:51.898007 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeacfffc_d5f1_4d0b_865c_ce5b4ae20a46.slice/crio-fcd7dd394555e97ce912708b4644098dbd72440ccd7e8be672239636ebc9d7f4 WatchSource:0}: Error finding container fcd7dd394555e97ce912708b4644098dbd72440ccd7e8be672239636ebc9d7f4: Status 404 returned error can't find the container with id fcd7dd394555e97ce912708b4644098dbd72440ccd7e8be672239636ebc9d7f4 Oct 30 22:05:51.900862 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.900828 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:51.900942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.900896 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:51.901020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.900941 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:51.954159 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:05:51.954133 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead1be97_52c5_489a_b659_f9342332885a.slice/crio-c0fc65bdd96b9b4309774dbfa289c63c1fec09da0b529ce0b45dccdf4cd67fa2 WatchSource:0}: Error finding container c0fc65bdd96b9b4309774dbfa289c63c1fec09da0b529ce0b45dccdf4cd67fa2: Status 404 returned error can't find the container with id c0fc65bdd96b9b4309774dbfa289c63c1fec09da0b529ce0b45dccdf4cd67fa2 Oct 30 22:05:51.956431 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.956410 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k"] Oct 30 22:05:51.957233 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.957189 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:51.957283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.957270 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:51.957403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:51.957335 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:52.097070 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.097029 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" event={"ID":"ead1be97-52c5-489a-b659-f9342332885a","Type":"ContainerStarted","Data":"bcf364773074fd9ec868256f27f0cea2eefa5df0eb8a37db831ecd20040e2364"} Oct 30 22:05:52.097295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.097098 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" event={"ID":"ead1be97-52c5-489a-b659-f9342332885a","Type":"ContainerStarted","Data":"c0fc65bdd96b9b4309774dbfa289c63c1fec09da0b529ce0b45dccdf4cd67fa2"} Oct 30 22:05:52.099132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.099105 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" event={"ID":"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46","Type":"ContainerStarted","Data":"c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd"} Oct 30 22:05:52.099238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.099153 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" event={"ID":"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46","Type":"ContainerStarted","Data":"fcd7dd394555e97ce912708b4644098dbd72440ccd7e8be672239636ebc9d7f4"} Oct 30 22:05:52.215627 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.215516 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podStartSLOduration=1.215497416 podStartE2EDuration="1.215497416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:52.212430145 +0000 UTC m=+1362.401967820" watchObservedRunningTime="2026-10-30 22:05:52.215497416 +0000 UTC m=+1362.405035095" Oct 30 22:05:52.329927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.329852 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podStartSLOduration=1.329833683 podStartE2EDuration="1.329833683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:52.32970072 +0000 UTC m=+1362.519238422" watchObservedRunningTime="2026-10-30 22:05:52.329833683 +0000 UTC m=+1362.519371360" Oct 30 22:05:52.447463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.447278 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:52.467996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.467891 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:05:52.469601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.469574 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:52.469723 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.469630 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:52.513280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:52.513244 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:05:53.468007 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:53.467975 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:53.468511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:53.468038 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:53.520331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:53.519788 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:53.520331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:53.519872 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:05:54.467751 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:54.467719 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:54.467901 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:54.467770 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:54.514860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:54.514737 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:54.514860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:54.514799 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:05:55.467791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:55.467758 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:55.467968 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:55.467813 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:55.513764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:55.513723 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:55.514049 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:55.514013 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:05:55.916553 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:55.916516 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:56.467896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:56.467865 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:56.468151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:56.467918 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:56.514805 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:56.514768 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:56.515085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:56.515061 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:05:57.467533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:57.467497 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:57.467926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:57.467558 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:57.513331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:57.512762 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:57.513331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:57.512826 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:05:58.421568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:58.421526 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:58.467598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:58.467554 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:58.468087 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:58.467611 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:58.512871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:58.512833 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:58.513029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:58.512880 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:05:59.468207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:59.468169 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 22:05:59.468713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:59.468225 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 22:05:59.513492 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:59.513463 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 22:05:59.513645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:05:59.513518 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" podUID="ead1be97-52c5-489a-b659-f9342332885a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 22:06:00.477150 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.477116 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:06:00.478060 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.478035 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:06:00.478888 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.478864 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:06:00.531255 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.531223 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:06:00.531658 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.531637 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:06:00.532722 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.532691 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k" Oct 30 22:06:00.843578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:00.843547 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:01.014107 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:01.013945 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:01.342217 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:01.342184 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:06.095922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.095879 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:06.282124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.282088 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt"] Oct 30 22:06:06.286679 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.286655 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334273 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334244 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334280 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334436 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334614 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334600 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334634 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbftb\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334687 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334659 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334702 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334901 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334888 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.334936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.334922 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.396505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.396431 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt"] Oct 30 22:06:06.437044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437001 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437234 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437212 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437266 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbftb\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437343 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437371 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437500 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437404 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437548 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437642 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437606 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437642 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437631 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437650 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437697 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.437985 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.437964 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.438056 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.438004 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.438759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.438732 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.440210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.440186 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.440443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.440427 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.500544 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.500512 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.500687 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.500584 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbftb\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:06.602830 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:06.602796 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:07.038371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.038274 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt"] Oct 30 22:06:07.074743 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:06:07.074715 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6043a15c_2b8e_44ae_af90_9f75919bc43c.slice/crio-276df4d1b9afe25b01b6becfc1cc0db1ab97ae9e3f25627b819e7669903c6577 WatchSource:0}: Error finding container 276df4d1b9afe25b01b6becfc1cc0db1ab97ae9e3f25627b819e7669903c6577: Status 404 returned error can't find the container with id 276df4d1b9afe25b01b6becfc1cc0db1ab97ae9e3f25627b819e7669903c6577 Oct 30 22:06:07.077635 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.077598 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:07.077762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.077680 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:07.077762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.077728 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:07.240329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.240267 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" event={"ID":"6043a15c-2b8e-44ae-af90-9f75919bc43c","Type":"ContainerStarted","Data":"20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee"} Oct 30 22:06:07.240916 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.240349 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" event={"ID":"6043a15c-2b8e-44ae-af90-9f75919bc43c","Type":"ContainerStarted","Data":"276df4d1b9afe25b01b6becfc1cc0db1ab97ae9e3f25627b819e7669903c6577"} Oct 30 22:06:07.360767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.360713 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podStartSLOduration=1.360699944 podStartE2EDuration="1.360699944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:07.356658002 +0000 UTC m=+1377.546195679" watchObservedRunningTime="2026-10-30 22:06:07.360699944 +0000 UTC m=+1377.550237618" Oct 30 22:06:07.610342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:07.603629 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:08.605173 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:08.605127 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:08.605608 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:08.605200 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:09.604127 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:09.604086 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:09.604373 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:09.604174 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:10.374044 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:06:10.374004 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:06:10.603291 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:10.603261 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:10.603496 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:10.603325 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:11.603922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:11.603880 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:11.604387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:11.603964 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:12.511997 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:12.511947 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:12.603860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:12.603829 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:12.604088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:12.604064 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:12.699870 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:12.699830 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:13.604034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:13.603999 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:13.604210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:13.604059 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:14.603624 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:14.603589 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 22:06:14.603797 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:14.603663 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 22:06:15.607587 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:15.607553 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:15.608049 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:15.607887 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:15.608526 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:15.608506 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:06:16.062787 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:16.062674 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:18.800433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:18.800391 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:06:19.067483 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.067381 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:06:19.074334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.072398 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.408455 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.408407 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.531643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.530420 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.570471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.570413 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.806403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.804458 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:06:19.806403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.805341 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:06:19.807810 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.807064 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.807810 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.807377 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:06:19.935206 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:19.935138 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.217441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.072552 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.217441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.078739 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.217441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.161100 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.500772 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.500663 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:06:20.566763 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.544587 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.596992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.595503 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:06:20.809333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.809177 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:06:20.810047 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.810015 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.810292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.810271 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.842710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.840848 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:20.842710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.840930 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.966944 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:20.966902 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.330127 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:06:21.330090 5508 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.381s" Oct 30 22:06:21.330632 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.330612 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:06:21.330857 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.330829 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:06:21.385472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.331064 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:06:21.385472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.331107 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:06:21.385472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.331142 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:06:21.385472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.331174 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:06:21.385472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.331208 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:06:21.385472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.331242 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:06:21.871598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.871557 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.876552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.876516 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.946513 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.925337 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.946513 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.926159 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.948036 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.947982 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:06:21.964417 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.948213 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:06:21.968474 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.968435 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.976376 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.972806 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:06:21.984599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.984566 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:06:22.060651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.985393 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:06:22.173387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:21.992448 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:22.173387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.163455 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:22.883332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.820686 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:06:22.883332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.879494 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:22.883332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.880053 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:06:22.993330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.970128 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:06:22.995867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.927553 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:06:22.996636 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.974396 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:22.997864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:22.980544 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.059429 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:23.059395 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:06:23.074375 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:23.065444 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:06:23.700048 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:23.700014 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:06:23.766213 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:23.766177 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.077626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.068361 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:24.077626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.076664 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.152330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.144274 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:06:24.152330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.145997 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:24.152330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.146250 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 22:06:24.155167 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.154234 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:06:24.155617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.155489 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:06:24.164471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.156925 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:06:24.201264 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.172116 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:06:24.373381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:24.331658 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:25.206331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.196535 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:25.229331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.225698 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 22:06:25.229331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.227552 5508 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-cvlq5 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.12:8443/healthz\": context deadline exceeded" start-of-body= Oct 30 22:06:25.229331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.227602 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cvlq5" podUID="6cb39913-45bd-4e03-8a53-e5a269238d4e" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.12:8443/healthz\": context deadline exceeded" Oct 30 22:06:25.229646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.229380 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 22:06:25.229646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.229635 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 22:06:25.236679 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.236630 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:06:25.260660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.259151 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:25.282481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.281953 5508 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:25.282481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.282032 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:25.303865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.286916 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:25.303865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.291620 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:25.337538 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:06:25.337506 5508 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.377s" Oct 30 22:06:25.337787 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.337767 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2"] Oct 30 22:06:25.363798 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.363773 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2"] Oct 30 22:06:25.364151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.364130 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:06:25.365698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.365677 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.429404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.429377 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:06:25.448343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.445855 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:06:25.453399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.453367 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:06:25.454511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.454488 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:06:25.454764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.454733 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbgf\" (UniqueName: \"kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.454858 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.454834 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.454929 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.454913 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.455054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.455032 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.462378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.462060 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:06:25.462378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.462116 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:06:25.462572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.462519 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:06:25.498720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.498675 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-97895160-serving-cert\"" Oct 30 22:06:25.557399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.557364 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbgf\" (UniqueName: \"kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.557600 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.557497 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.557600 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.557580 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.557745 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.557716 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.558856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.558829 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.561082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.561058 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.561160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.561083 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.625737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.625704 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbgf\" (UniqueName: \"kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf\") pod \"maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:25.767458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:25.767375 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:26.203977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:26.203942 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2"] Oct 30 22:06:26.208844 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:06:26.208769 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bc7cb4_53b8_46aa_85a7_525f7757dde1.slice/crio-d3923f1e389e68898c282e320ea8ad20c403cafebde1f8cd8a9f507a155bcda8 WatchSource:0}: Error finding container d3923f1e389e68898c282e320ea8ad20c403cafebde1f8cd8a9f507a155bcda8: Status 404 returned error can't find the container with id d3923f1e389e68898c282e320ea8ad20c403cafebde1f8cd8a9f507a155bcda8 Oct 30 22:06:26.210744 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:26.210721 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:06:26.422764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:26.422732 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" event={"ID":"96bc7cb4-53b8-46aa-85a7-525f7757dde1","Type":"ContainerStarted","Data":"d3923f1e389e68898c282e320ea8ad20c403cafebde1f8cd8a9f507a155bcda8"} Oct 30 22:06:27.433038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:27.433006 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" event={"ID":"96bc7cb4-53b8-46aa-85a7-525f7757dde1","Type":"ContainerStarted","Data":"3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1"} Oct 30 22:06:27.433448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:27.433072 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:27.553853 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:27.553787 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" podStartSLOduration=3.806705825 podStartE2EDuration="4.553766434s" totalImagesPullingTime="747.060609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:27.548866408 +0000 UTC m=+1397.738404109" watchObservedRunningTime="2026-10-30 22:06:27.553766434 +0000 UTC m=+1397.743304110" Oct 30 22:06:28.844911 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:28.844866 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:29.920825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:29.920788 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.444803 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:33.444773 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:06:34.772886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:34.772848 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.697054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:39.696997 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:40.374898 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:06:40.374857 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:06:44.567932 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:44.567902 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:49.374120 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:49.374083 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:54.161507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:54.161465 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:59.132691 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:06:59.132652 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:04.645671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:04.645634 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:10.171557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:10.171526 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:10.374235 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:07:10.374201 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:07:15.745612 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:15.745581 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.559610 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:18.559520 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.717107 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:18.717073 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:19.730335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:19.730025 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:20.389111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:20.389073 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:21.817611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:21.817560 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:27.349659 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:27.349582 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:29.064187 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:29.064154 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:29.226331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:29.226266 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:33.217942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:33.217903 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:33.921117 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:33.921080 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:38.560283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:38.560245 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:40.214356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:40.214297 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:40.373224 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:07:40.373173 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:07:43.812182 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:43.812142 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:44.326894 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:44.326831 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:44.485199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:44.485127 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:48.678634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:48.678596 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:53.466026 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:53.465988 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:55.827828 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:55.827792 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.777924 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:57.777881 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:59.554378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:59.554334 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:59.790169 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:59.790135 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:59.947355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:07:59.947294 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:00.270507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:00.270399 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:00.451327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:00.449999 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:00.954871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:00.954833 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.122984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:01.122951 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.292159 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:01.292077 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:02.038175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:02.038098 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:02.402377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:02.402339 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:04.733920 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:04.733886 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.140485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:06.140444 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.346427 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:06.346394 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.954285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:06.954249 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:07.731964 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:07.731929 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:08.019617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:08.019533 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:08.662608 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:08.662575 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:08.849083 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:08.849047 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:09.421550 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:09.421505 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:10.296093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:10.295819 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:10.428534 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:08:10.428499 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:08:10.678942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:10.678769 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:12.217330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:12.216822 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2"] Oct 30 22:08:12.217330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:12.217061 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" podUID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" containerName="maas-api" containerID="cri-o://3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1" gracePeriod=30 Oct 30 22:08:12.674026 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:12.673985 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:12.955387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:12.952065 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:12.981331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:12.966410 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:13.073436 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.073388 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:13.098013 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.097422 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:13.129208 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.129169 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:13.494005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.493655 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:08:13.502563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.502021 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls\") pod \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " Oct 30 22:08:13.502563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.502131 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs\") pod \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " Oct 30 22:08:13.502563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.502224 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca\") pod \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " Oct 30 22:08:13.502563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.502270 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf\" (UniqueName: \"kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf\") pod \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\" (UID: \"96bc7cb4-53b8-46aa-85a7-525f7757dde1\") " Oct 30 22:08:13.505829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.505799 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca" pod "96bc7cb4-53b8-46aa-85a7-525f7757dde1" (UID: "96bc7cb4-53b8-46aa-85a7-525f7757dde1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:13.529407 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.529348 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs" pod "96bc7cb4-53b8-46aa-85a7-525f7757dde1" (UID: "96bc7cb4-53b8-46aa-85a7-525f7757dde1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:13.529759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.529695 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls" pod "96bc7cb4-53b8-46aa-85a7-525f7757dde1" (UID: "96bc7cb4-53b8-46aa-85a7-525f7757dde1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:13.531319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.531269 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf" pod "96bc7cb4-53b8-46aa-85a7-525f7757dde1" (UID: "96bc7cb4-53b8-46aa-85a7-525f7757dde1"). InnerVolumeSpecName "kube-api-access-qvbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:13.612332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.611935 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc7cb4-53b8-46aa-85a7-525f7757dde1-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:13.612332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.611972 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbgf\" (UniqueName: \"kubernetes.io/projected/96bc7cb4-53b8-46aa-85a7-525f7757dde1-kube-api-access-qvbgf\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:13.612332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.611997 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:13.612332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.612012 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96bc7cb4-53b8-46aa-85a7-525f7757dde1-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:13.892329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.891089 5508 generic.go:356] "Generic (PLEG): container finished" podID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" containerID="3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1" exitCode=0 Oct 30 22:08:13.892329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.891286 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" event={"ID":"96bc7cb4-53b8-46aa-85a7-525f7757dde1","Type":"ContainerDied","Data":"3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1"} Oct 30 22:08:13.892329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.891394 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" event={"ID":"96bc7cb4-53b8-46aa-85a7-525f7757dde1","Type":"ContainerDied","Data":"d3923f1e389e68898c282e320ea8ad20c403cafebde1f8cd8a9f507a155bcda8"} Oct 30 22:08:13.892329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.891423 5508 scope.go:122] "RemoveContainer" containerID="3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1" Oct 30 22:08:13.892329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.891586 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" Oct 30 22:08:13.922788 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.922518 5508 scope.go:122] "RemoveContainer" containerID="3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1" Oct 30 22:08:13.922994 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:08:13.922970 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1\": container with ID starting with 3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1 not found: ID does not exist" containerID="3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1" Oct 30 22:08:13.923067 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:13.923006 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1"} err="failed to get container status \"3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1\": rpc error: code = NotFound desc = could not find container \"3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1\": container with ID starting with 3af286e32e3e15ab6a0556e0841898b88f223eff78b52ebe52051b7dcd9fd9e1 not found: ID does not exist" Oct 30 22:08:14.064587 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:14.064551 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2"] Oct 30 22:08:14.067845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:14.067818 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2"] Oct 30 22:08:14.622368 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:14.622328 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:14.892067 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:14.892027 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.056873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:15.056780 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.532741 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:15.532704 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.762089 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:15.762044 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.953854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:15.953820 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" path="/var/lib/kubelet/pods/96bc7cb4-53b8-46aa-85a7-525f7757dde1/volumes" Oct 30 22:08:16.445238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:16.445192 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-97895160-6896c9c5cd-tmbl2" podUID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.81:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:16.719423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:16.718813 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.609982 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:17.609944 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.789398 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:17.789362 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:18.961752 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:18.961722 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:19.145343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:19.141270 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:20.684876 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:20.684842 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:26.026332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:26.024357 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:28.912569 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:28.912533 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:29.601037 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:29.601002 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:31.110677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:31.109924 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:34.353393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:34.353356 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:36.258430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:36.258290 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:37.092325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:37.092198 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:37.385232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:37.385152 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:40.375806 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:08:40.375772 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:08:41.158137 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:41.158090 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:41.397607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:41.397575 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.372726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:44.372682 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.542942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:46.542892 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.976349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:46.976294 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:48.959468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:48.959256 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:49.710629 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:49.710595 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:50.052207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:50.052092 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:52.139382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:52.139294 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:52.387120 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:52.387085 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:52.630975 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:52.630942 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:53.818425 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:53.818386 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:55.080944 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:55.080902 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:55.577974 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:55.577906 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:57.882338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:57.882274 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.524855 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.524819 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.810757 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.810667 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7"] Oct 30 22:08:58.811618 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.811588 5508 cpu_manager.go:414] "containerMap: removing container" podUID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" containerName="maas-api" Oct 30 22:08:58.811618 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.811618 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" containerName="maas-api" Oct 30 22:08:58.811835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.811788 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bc7cb4-53b8-46aa-85a7-525f7757dde1" containerName="maas-api" Oct 30 22:08:58.816251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.816217 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:58.843903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.843166 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.902477 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.899631 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:58.902477 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.899769 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xkw\" (UniqueName: \"kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw\") pod \"payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:58.933366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.924609 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2a25a4-dockercfg-sdwxh\"" Oct 30 22:08:58.933366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.924750 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2a25a4\"" Oct 30 22:08:58.983335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:58.973672 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7"] Oct 30 22:08:59.007084 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.001588 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:59.007451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.003742 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:59.007651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.007629 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xkw\" (UniqueName: \"kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw\") pod \"payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:59.073484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.073451 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xkw\" (UniqueName: \"kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw\") pod \"payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:59.147199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.147161 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:08:59.163412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.161392 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:59.546674 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:08:59.546638 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7"] Oct 30 22:08:59.548704 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:08:59.548675 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38fcb34_e5d9_4153_a18e_167062d68e7b.slice/crio-e6eb94b9ebe588496d3b246df1748383dee876b3682e3243d1582f7bf25a8cb3 WatchSource:0}: Error finding container e6eb94b9ebe588496d3b246df1748383dee876b3682e3243d1582f7bf25a8cb3: Status 404 returned error can't find the container with id e6eb94b9ebe588496d3b246df1748383dee876b3682e3243d1582f7bf25a8cb3 Oct 30 22:09:00.264139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:00.264104 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" event={"ID":"b38fcb34-e5d9-4153-a18e-167062d68e7b","Type":"ContainerStarted","Data":"9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7"} Oct 30 22:09:00.264581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:00.264166 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" event={"ID":"b38fcb34-e5d9-4153-a18e-167062d68e7b","Type":"ContainerStarted","Data":"e6eb94b9ebe588496d3b246df1748383dee876b3682e3243d1582f7bf25a8cb3"} Oct 30 22:09:00.264581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:00.264196 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:09:00.386940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:00.386886 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" podStartSLOduration=2.386872043 podStartE2EDuration="2.386872043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:00.383385932 +0000 UTC m=+1550.572923616" watchObservedRunningTime="2026-10-30 22:09:00.386872043 +0000 UTC m=+1550.576409718" Oct 30 22:09:03.298218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.298180 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r"] Oct 30 22:09:03.302595 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.302579 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.405562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.405527 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-497924bf-serving-cert\"" Oct 30 22:09:03.409234 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.409209 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r"] Oct 30 22:09:03.416252 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.416215 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.416480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.416458 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bg4d\" (UniqueName: \"kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.416665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.416644 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.416734 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.416704 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.518711 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.518677 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.518896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.518813 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bg4d\" (UniqueName: \"kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.518956 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.518930 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.519014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.518961 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.519125 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:09:03.519112 5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-serving-cert: secret "maas-api-e2e-derive-497924bf-serving-cert" not found Oct 30 22:09:03.519177 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:09:03.519173 5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls podName:80380e39-4926-405e-a689-9aea39dd12ec nodeName:}" failed. No retries permitted until 2026-10-30 22:09:04.019152221 +0000 UTC m=+1554.208689875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls") pod "maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" (UID: "80380e39-4926-405e-a689-9aea39dd12ec") : secret "maas-api-e2e-derive-497924bf-serving-cert" not found Oct 30 22:09:03.519620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.519600 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.521356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.521338 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.527764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.527731 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:03.582404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.582364 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bg4d\" (UniqueName: \"kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:03.727340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.727272 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:03.919778 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:03.919686 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.028505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:04.028474 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:04.031271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:04.031248 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\") pod \"maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:04.123085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:04.123046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.222987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:04.222908 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:04.594347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:04.594267 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.608619 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:09:04.608593 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80380e39_4926_405e_a689_9aea39dd12ec.slice/crio-4d5068e5b7fdd62970648758738110d480975d941d9c6adc5a8d55bdd6418225 WatchSource:0}: Error finding container 4d5068e5b7fdd62970648758738110d480975d941d9c6adc5a8d55bdd6418225: Status 404 returned error can't find the container with id 4d5068e5b7fdd62970648758738110d480975d941d9c6adc5a8d55bdd6418225 Oct 30 22:09:04.611677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:04.611655 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r"] Oct 30 22:09:05.305497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:05.305460 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" event={"ID":"80380e39-4926-405e-a689-9aea39dd12ec","Type":"ContainerStarted","Data":"4d5068e5b7fdd62970648758738110d480975d941d9c6adc5a8d55bdd6418225"} Oct 30 22:09:06.314132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:06.314091 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" event={"ID":"80380e39-4926-405e-a689-9aea39dd12ec","Type":"ContainerStarted","Data":"bdba92ee234edf191a20e9b3d70b494f44bd55c6f158088759494df9bc7b5519"} Oct 30 22:09:06.314643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:06.314188 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:06.429960 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:06.429901 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" podStartSLOduration=2.6258768569999997 podStartE2EDuration="3.429886426s" totalImagesPullingTime="804.009569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:06.427476079 +0000 UTC m=+1556.617013754" watchObservedRunningTime="2026-10-30 22:09:06.429886426 +0000 UTC m=+1556.619424100" Oct 30 22:09:08.874510 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:08.874471 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:10.374220 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:09:10.374182 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:09:11.275773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:11.275736 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:09:11.910638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:11.910577 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:12.108892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:12.108860 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:12.327011 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:12.326984 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:09:13.859158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:13.859126 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:18.787370 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:18.787335 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.583328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:19.583268 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.766452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:19.766418 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:23.694946 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:23.694915 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:28.589852 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:28.589813 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:29.998239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:29.998187 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:34.045809 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:34.045776 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:34.416563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:34.416529 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:35.275033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:35.274995 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:35.550877 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:35.550793 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:39.468149 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:39.468116 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.008875 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:40.008842 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.195138 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:40.195105 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.374167 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:09:40.374133 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:09:40.390408 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:40.390382 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:43.270641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:43.270599 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:43.450041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:43.449892 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:45.417562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:45.417526 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:50.137767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.137731 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5"] Oct 30 22:09:50.142648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.142629 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.195111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.195078 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-2a25a4-serving-cert\"" Oct 30 22:09:50.248354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.248299 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5"] Oct 30 22:09:50.307342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.307282 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.307539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.307369 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.307539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.307453 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.307539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.307506 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9x7x\" (UniqueName: \"kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.368047 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.367999 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:50.426240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.417479 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.426240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.425147 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.426240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.425245 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.426240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.425347 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9x7x\" (UniqueName: \"kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.431150 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.431099 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.431873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.431827 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.432787 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.432763 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.483682 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.483655 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9x7x\" (UniqueName: \"kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x\") pod \"maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.636396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.636361 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:50.756652 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.756550 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:50.817737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:50.817672 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:51.152730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:51.152698 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5"] Oct 30 22:09:51.154064 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:09:51.154030 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ebe6c5_3c70_43ba_8a54_060dbf58390a.slice/crio-22e6f40eaafdeace10444680c408651a08f5efb2ba56a165c5974d95b1db19df WatchSource:0}: Error finding container 22e6f40eaafdeace10444680c408651a08f5efb2ba56a165c5974d95b1db19df: Status 404 returned error can't find the container with id 22e6f40eaafdeace10444680c408651a08f5efb2ba56a165c5974d95b1db19df Oct 30 22:09:51.681926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:51.681892 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" event={"ID":"85ebe6c5-3c70-43ba-8a54-060dbf58390a","Type":"ContainerStarted","Data":"22e6f40eaafdeace10444680c408651a08f5efb2ba56a165c5974d95b1db19df"} Oct 30 22:09:52.690064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:52.690021 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" event={"ID":"85ebe6c5-3c70-43ba-8a54-060dbf58390a","Type":"ContainerStarted","Data":"7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c"} Oct 30 22:09:52.690464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:52.690092 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:52.808648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:52.808597 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" podStartSLOduration=1.990517085 podStartE2EDuration="2.808583907s" totalImagesPullingTime="818.066822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:52.804646771 +0000 UTC m=+1602.994184488" watchObservedRunningTime="2026-10-30 22:09:52.808583907 +0000 UTC m=+1602.998121608" Oct 30 22:09:56.025042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:56.025008 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:58.703683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:58.703655 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:09:59.080927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:59.080885 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.261349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:59.261299 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.524075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:09:59.523972 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:00.061852 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:00.061818 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:01.464823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:01.464788 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.623437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:04.623396 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.791873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:04.791820 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:05.537806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:05.537765 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:05.861511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:05.861461 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:07.153180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:07.153143 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:10.375124 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:10:10.375087 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:10:11.382414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:11.382379 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.623753 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:11.623717 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.910771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:11.910741 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:12.588683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:12.588647 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:14.240167 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:14.240133 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw"] Oct 30 22:10:14.240643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:14.240383 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" containerID="cri-o://e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833" gracePeriod=30 Oct 30 22:10:14.288766 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:14.288735 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw"] Oct 30 22:10:14.288965 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:14.288945 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" containerID="cri-o://0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1" gracePeriod=30 Oct 30 22:10:14.594996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:14.594964 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.461751 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:15.461713 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.777836 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:15.777656 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:17.136725 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:17.136687 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:17.331727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:17.331653 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:18.233740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:18.233598 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:18.929491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:18.929454 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.154491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.154461 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.653938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.653911 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:10:19.657329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.657292 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:10:19.706274 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706250 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706424 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706487 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706536 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.706667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706650 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706714 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706687 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706782 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706762 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.706843 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706786 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706843 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706814 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706896 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.706945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706908 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.707049 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.706926 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk\") pod \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\" (UID: \"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f\") " Oct 30 22:10:19.707222 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.707176 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:19.707222 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.707299 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.708132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.708032 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.708132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.708060 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.708132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.708075 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.708132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.708092 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.708132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.708107 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.709355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.709328 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "kube-api-access-tbcvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:19.709533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.709512 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.709589 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.709550 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:19.709589 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.709563 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo" pod "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" (UID: "a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:19.809238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809168 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809224 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809278 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809344 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809394 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809451 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809663 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809611 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809663 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809641 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.809773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809689 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.809773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809717 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809718 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:19.809773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809758 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy\") pod \"9eca57e9-2749-4ccb-9266-5261b4772220\" (UID: \"9eca57e9-2749-4ccb-9266-5261b4772220\") " Oct 30 22:10:19.809963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.809860 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.810043 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810008 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.810754 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810727 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.810840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810759 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eca57e9-2749-4ccb-9266-5261b4772220-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.810840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810776 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.810840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810790 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.810840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810805 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.810840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810822 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.810840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810836 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.811057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810851 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.811057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.810866 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbcvk\" (UniqueName: \"kubernetes.io/projected/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f-kube-api-access-tbcvk\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.812161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.812140 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:19.812398 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.812370 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "kube-api-access-sqvck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:19.812454 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.812404 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:19.812525 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.812503 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo" pod "9eca57e9-2749-4ccb-9266-5261b4772220" (UID: "9eca57e9-2749-4ccb-9266-5261b4772220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:19.912512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.912479 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.912512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.912506 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvck\" (UniqueName: \"kubernetes.io/projected/9eca57e9-2749-4ccb-9266-5261b4772220-kube-api-access-sqvck\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.912512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.912519 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eca57e9-2749-4ccb-9266-5261b4772220-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.912735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.912528 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eca57e9-2749-4ccb-9266-5261b4772220-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:19.983348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.983299 5508 generic.go:356] "Generic (PLEG): container finished" podID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerID="e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833" exitCode=0 Oct 30 22:10:19.983465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.983359 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" event={"ID":"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f","Type":"ContainerDied","Data":"e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833"} Oct 30 22:10:19.983465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.983387 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" Oct 30 22:10:19.983465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.983423 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw" event={"ID":"a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f","Type":"ContainerDied","Data":"b5ebde2931f9e1acbd532849b72c3a54333838473dd22c352c1b5c643162f513"} Oct 30 22:10:19.983465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.983457 5508 scope.go:122] "RemoveContainer" containerID="e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833" Oct 30 22:10:19.985478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.985448 5508 generic.go:356] "Generic (PLEG): container finished" podID="9eca57e9-2749-4ccb-9266-5261b4772220" containerID="0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1" exitCode=0 Oct 30 22:10:19.985588 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.985479 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" event={"ID":"9eca57e9-2749-4ccb-9266-5261b4772220","Type":"ContainerDied","Data":"0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1"} Oct 30 22:10:19.985588 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.985502 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" Oct 30 22:10:19.985588 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.985545 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw" event={"ID":"9eca57e9-2749-4ccb-9266-5261b4772220","Type":"ContainerDied","Data":"f0539634dc625dfa7b1be5e35ebb84af2e6781530f4448c843999c763eb96d71"} Oct 30 22:10:19.996528 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.996505 5508 scope.go:122] "RemoveContainer" containerID="e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833" Oct 30 22:10:19.996843 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:10:19.996819 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833\": container with ID starting with e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833 not found: ID does not exist" containerID="e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833" Oct 30 22:10:19.996938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.996863 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833"} err="failed to get container status \"e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833\": rpc error: code = NotFound desc = could not find container \"e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833\": container with ID starting with e76fd3d3a4203191892dabb4df5b82edb89b6e94caedf2734f9de5de2148a833 not found: ID does not exist" Oct 30 22:10:19.996938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:19.996889 5508 scope.go:122] "RemoveContainer" containerID="0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1" Oct 30 22:10:20.010068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:20.010049 5508 scope.go:122] "RemoveContainer" containerID="0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1" Oct 30 22:10:20.011058 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:10:20.010741 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1\": container with ID starting with 0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1 not found: ID does not exist" containerID="0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1" Oct 30 22:10:20.011058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:20.010780 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1"} err="failed to get container status \"0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1\": rpc error: code = NotFound desc = could not find container \"0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1\": container with ID starting with 0d8bb8d2e7cce5a6b54157663ce7abb5e490c1c7ad780da9ac06028276e0bed1 not found: ID does not exist" Oct 30 22:10:20.117951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:20.117916 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw"] Oct 30 22:10:20.126132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:20.126099 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-v6bmw"] Oct 30 22:10:20.244787 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:20.244748 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw"] Oct 30 22:10:20.254021 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:20.253985 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-tfntw"] Oct 30 22:10:21.071292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:21.071250 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.275175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:21.275143 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.507672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:21.507577 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.955233 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:21.955204 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" path="/var/lib/kubelet/pods/9eca57e9-2749-4ccb-9266-5261b4772220/volumes" Oct 30 22:10:21.955704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:21.955689 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" path="/var/lib/kubelet/pods/a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f/volumes" Oct 30 22:10:24.350255 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:24.350223 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:24.564331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:24.563011 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:24.757005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:24.756920 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:26.452127 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:26.452085 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.190555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:28.190495 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.374003 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:28.370379 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:30.263638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:30.263593 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:30.654694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:30.651815 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:32.353143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:32.353108 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:35.571420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:35.571381 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:40.374196 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:10:40.374157 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:10:40.464529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:40.464498 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:40.671879 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:40.671790 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:45.666639 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:45.666583 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:49.932508 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:49.932475 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.499426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:51.499385 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.814167 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:51.814079 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.982963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:51.982924 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.171348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:52.171279 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.952165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:52.952129 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:53.593939 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:53.593906 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:53.972854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:53.972771 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:57.615180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:57.615125 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.184265 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:58.184230 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx"] Oct 30 22:10:58.184533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:58.184502 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" containerID="cri-o://0954ee3127d8945a33df82e25d65e9edcb8d41babcdfa55153bff38ffe3a4920" gracePeriod=30 Oct 30 22:10:58.503780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:58.503708 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7"] Oct 30 22:10:58.503959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:58.503918 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" containerID="cri-o://619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6" gracePeriod=30 Oct 30 22:10:58.864585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:58.864552 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv"] Oct 30 22:10:58.865010 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:58.864749 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" containerID="cri-o://e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543" gracePeriod=30 Oct 30 22:10:59.032818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:59.032781 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.224225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:59.224142 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.404936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:10:59.404895 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.127821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.127788 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r"] Oct 30 22:11:02.128357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.128043 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" podUID="80380e39-4926-405e-a689-9aea39dd12ec" containerName="maas-api" containerID="cri-o://bdba92ee234edf191a20e9b3d70b494f44bd55c6f158088759494df9bc7b5519" gracePeriod=30 Oct 30 22:11:02.258191 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.258158 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.338045 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.338005 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" podUID="80380e39-4926-405e-a689-9aea39dd12ec" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.83:8443/health\": dial tcp 10.134.0.83:8443: connect: connection refused" Oct 30 22:11:02.411217 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.411139 5508 generic.go:356] "Generic (PLEG): container finished" podID="80380e39-4926-405e-a689-9aea39dd12ec" containerID="bdba92ee234edf191a20e9b3d70b494f44bd55c6f158088759494df9bc7b5519" exitCode=0 Oct 30 22:11:02.411384 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.411271 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" event={"ID":"80380e39-4926-405e-a689-9aea39dd12ec","Type":"ContainerDied","Data":"bdba92ee234edf191a20e9b3d70b494f44bd55c6f158088759494df9bc7b5519"} Oct 30 22:11:02.426470 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.426366 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.487390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.487285 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:11:02.569884 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.569834 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d\" (UniqueName: \"kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d\") pod \"80380e39-4926-405e-a689-9aea39dd12ec\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " Oct 30 22:11:02.570539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.570514 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\") pod \"80380e39-4926-405e-a689-9aea39dd12ec\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " Oct 30 22:11:02.580794 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.580760 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca\") pod \"80380e39-4926-405e-a689-9aea39dd12ec\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " Oct 30 22:11:02.581067 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.581041 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs\") pod \"80380e39-4926-405e-a689-9aea39dd12ec\" (UID: \"80380e39-4926-405e-a689-9aea39dd12ec\") " Oct 30 22:11:02.591822 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.591692 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca" pod "80380e39-4926-405e-a689-9aea39dd12ec" (UID: "80380e39-4926-405e-a689-9aea39dd12ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:02.592517 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.592473 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls" pod "80380e39-4926-405e-a689-9aea39dd12ec" (UID: "80380e39-4926-405e-a689-9aea39dd12ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:02.593331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.593277 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs" pod "80380e39-4926-405e-a689-9aea39dd12ec" (UID: "80380e39-4926-405e-a689-9aea39dd12ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:02.603273 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.598001 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d" pod "80380e39-4926-405e-a689-9aea39dd12ec" (UID: "80380e39-4926-405e-a689-9aea39dd12ec"). InnerVolumeSpecName "kube-api-access-9bg4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:02.688591 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.687514 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.690099 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.690078 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80380e39-4926-405e-a689-9aea39dd12ec-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.690179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.690102 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.690179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.690116 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bg4d\" (UniqueName: \"kubernetes.io/projected/80380e39-4926-405e-a689-9aea39dd12ec-kube-api-access-9bg4d\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.690179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.690130 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80380e39-4926-405e-a689-9aea39dd12ec-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:02.851332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:02.851260 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.030420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.029577 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.576735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.576647 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz"] Oct 30 22:11:03.578729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.578702 5508 cpu_manager.go:414] "containerMap: removing container" podUID="80380e39-4926-405e-a689-9aea39dd12ec" containerName="maas-api" Oct 30 22:11:03.578932 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.578916 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80380e39-4926-405e-a689-9aea39dd12ec" containerName="maas-api" Oct 30 22:11:03.579108 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.579094 5508 cpu_manager.go:414] "containerMap: removing container" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" Oct 30 22:11:03.579210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.579197 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" Oct 30 22:11:03.579487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.579470 5508 cpu_manager.go:414] "containerMap: removing container" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" Oct 30 22:11:03.579592 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.579579 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" Oct 30 22:11:03.579909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.579891 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="80380e39-4926-405e-a689-9aea39dd12ec" containerName="maas-api" Oct 30 22:11:03.580018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.580007 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66b2e30-b7ec-4006-a6fe-8f5b288d9c5f" containerName="istio-proxy" Oct 30 22:11:03.580109 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.580099 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eca57e9-2749-4ccb-9266-5261b4772220" containerName="istio-proxy" Oct 30 22:11:03.596560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.593509 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.596560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.596441 5508 generic.go:356] "Generic (PLEG): container finished" podID="0527abff-b56e-457a-a62f-cae88f284eea" containerID="0954ee3127d8945a33df82e25d65e9edcb8d41babcdfa55153bff38ffe3a4920" exitCode=0 Oct 30 22:11:03.598420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.598393 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz"] Oct 30 22:11:03.598778 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.598748 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" event={"ID":"0527abff-b56e-457a-a62f-cae88f284eea","Type":"ContainerDied","Data":"0954ee3127d8945a33df82e25d65e9edcb8d41babcdfa55153bff38ffe3a4920"} Oct 30 22:11:03.599107 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.599083 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" Oct 30 22:11:03.600144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.598672 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.600421 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.599219 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r" event={"ID":"80380e39-4926-405e-a689-9aea39dd12ec","Type":"ContainerDied","Data":"4d5068e5b7fdd62970648758738110d480975d941d9c6adc5a8d55bdd6418225"} Oct 30 22:11:03.600580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.600558 5508 scope.go:122] "RemoveContainer" containerID="bdba92ee234edf191a20e9b3d70b494f44bd55c6f158088759494df9bc7b5519" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666221 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666284 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwc7\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666345 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666390 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666503 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666551 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666578 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666655 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.671927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.666750 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.682489 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.679056 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7c826c-openshift-default-dockercfg-k9p7f\"" Oct 30 22:11:03.701695 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.700482 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:11:03.701695 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.700549 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.772838 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.772908 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.772935 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773010 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773107 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773208 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773242 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwc7\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773271 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773340 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.773892 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.774166 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.774469 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.775351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.774724 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.787775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.777832 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.801024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.796580 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.808281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.808223 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.891552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.886167 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwc7\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.891552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.891162 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token\") pod \"e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:03.972009 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.971978 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:11:03.972481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.972011 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r"] Oct 30 22:11:03.972617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.972608 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-497924bf-78b9747dd9-pwl8r"] Oct 30 22:11:03.999851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:03.999828 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:11:04.047190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.012520 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:04.047190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.030547 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:11:04.047190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.030626 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:11:04.117341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.116953 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119174 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119237 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119295 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119381 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119460 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119548 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119611 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119656 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119748 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119815 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119901 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.119961 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.120050 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.120181 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert\") pod \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\" (UID: \"03ecca1a-530c-4f48-9e42-a7d7408ba81f\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.120261 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.120344 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.120505 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.126347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.120573 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh\") pod \"0527abff-b56e-457a-a62f-cae88f284eea\" (UID: \"0527abff-b56e-457a-a62f-cae88f284eea\") " Oct 30 22:11:04.133224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.130443 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.133224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.130786 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.136207 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.136558 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.136779 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.136984 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.137412 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.137783 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.138575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.138108 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.141865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.141083 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:04.144503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.144294 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:04.144503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.144481 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.144736 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.144703 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:04.145035 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.145007 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.145501 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.145466 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.147081 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.147057 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "kube-api-access-295dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.147935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.147900 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy" pod "0527abff-b56e-457a-a62f-cae88f284eea" (UID: "0527abff-b56e-457a-a62f-cae88f284eea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.155117 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.155073 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68" pod "03ecca1a-530c-4f48-9e42-a7d7408ba81f" (UID: "03ecca1a-530c-4f48-9e42-a7d7408ba81f"). InnerVolumeSpecName "kube-api-access-7pt68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.226370 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.224901 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.224945 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226618 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0527abff-b56e-457a-a62f-cae88f284eea-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226654 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226668 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226681 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226695 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ecca1a-530c-4f48-9e42-a7d7408ba81f-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226708 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226721 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226735 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0527abff-b56e-457a-a62f-cae88f284eea-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226749 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226762 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226776 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-295dh\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-kube-api-access-295dh\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226790 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226804 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pt68\" (UniqueName: \"kubernetes.io/projected/03ecca1a-530c-4f48-9e42-a7d7408ba81f-kube-api-access-7pt68\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226819 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0527abff-b56e-457a-a62f-cae88f284eea-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226832 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0527abff-b56e-457a-a62f-cae88f284eea-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.226867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.226844 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ecca1a-530c-4f48-9e42-a7d7408ba81f-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.357691 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.357662 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:11:04.381919 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.381884 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:04.434854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.434761 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.434854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.434835 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.435077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.434894 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.435077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.434965 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.435077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435003 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.435396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435255 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.435396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435338 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.435396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435372 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.436345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435403 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.436345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435616 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs\") pod \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\" (UID: \"03ae515e-2442-4885-b4d9-d1d54a2d8eda\") " Oct 30 22:11:04.436345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435609 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.436345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.435967 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.436345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.436081 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:04.436345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.436160 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.436835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.436653 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.436835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.436678 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.436835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.436690 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.436835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.436703 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.437776 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.437734 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.438099 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.438070 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:04.438401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.438379 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.438792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.438772 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l" pod "03ae515e-2442-4885-b4d9-d1d54a2d8eda" (UID: "03ae515e-2442-4885-b4d9-d1d54a2d8eda"). InnerVolumeSpecName "kube-api-access-nwz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.538240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.538205 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.538240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.538238 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.538576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.538251 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.538576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.538264 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ae515e-2442-4885-b4d9-d1d54a2d8eda-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.538576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.538276 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwz4l\" (UniqueName: \"kubernetes.io/projected/03ae515e-2442-4885-b4d9-d1d54a2d8eda-kube-api-access-nwz4l\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.541940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.541911 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz"] Oct 30 22:11:04.543429 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:11:04.543402 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323cc524_025b_4d6d_bf45_845ac6156590.slice/crio-e885690c8025e59253f18923c9878a3f7f867c280b25c00c17b1da83cf9049be WatchSource:0}: Error finding container e885690c8025e59253f18923c9878a3f7f867c280b25c00c17b1da83cf9049be: Status 404 returned error can't find the container with id e885690c8025e59253f18923c9878a3f7f867c280b25c00c17b1da83cf9049be Oct 30 22:11:04.546038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.545998 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:04.546157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.546067 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:04.546157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.546093 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:04.613981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.613946 5508 generic.go:356] "Generic (PLEG): container finished" podID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerID="619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6" exitCode=0 Oct 30 22:11:04.614444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.613991 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" Oct 30 22:11:04.614444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.614027 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" event={"ID":"03ecca1a-530c-4f48-9e42-a7d7408ba81f","Type":"ContainerDied","Data":"619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6"} Oct 30 22:11:04.614444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.614088 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7" event={"ID":"03ecca1a-530c-4f48-9e42-a7d7408ba81f","Type":"ContainerDied","Data":"7a6f57e737c802695c76cac9942df66dbe8be7cdecd2ac62e3cb95c7f54eb37c"} Oct 30 22:11:04.614444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.614118 5508 scope.go:122] "RemoveContainer" containerID="619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6" Oct 30 22:11:04.616021 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.615992 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" event={"ID":"0527abff-b56e-457a-a62f-cae88f284eea","Type":"ContainerDied","Data":"08dd7d09b7bbf4a2c4c2845f0bf33f1aecabbb2002589a01b819a468a63a258f"} Oct 30 22:11:04.616115 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.615997 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx" Oct 30 22:11:04.619088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.619034 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" event={"ID":"323cc524-025b-4d6d-bf45-845ac6156590","Type":"ContainerStarted","Data":"e885690c8025e59253f18923c9878a3f7f867c280b25c00c17b1da83cf9049be"} Oct 30 22:11:04.621378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.621353 5508 generic.go:356] "Generic (PLEG): container finished" podID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerID="e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543" exitCode=0 Oct 30 22:11:04.621476 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.621448 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" event={"ID":"03ae515e-2442-4885-b4d9-d1d54a2d8eda","Type":"ContainerDied","Data":"e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543"} Oct 30 22:11:04.621540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.621487 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" event={"ID":"03ae515e-2442-4885-b4d9-d1d54a2d8eda","Type":"ContainerDied","Data":"8f4654edeb7ecd89a521ba0b4fec3236b1eff628aa622cb22358afc50aba87ef"} Oct 30 22:11:04.621609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.621554 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv" Oct 30 22:11:04.627095 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.627073 5508 scope.go:122] "RemoveContainer" containerID="619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6" Oct 30 22:11:04.627414 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:04.627393 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6\": container with ID starting with 619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6 not found: ID does not exist" containerID="619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6" Oct 30 22:11:04.627510 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.627425 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6"} err="failed to get container status \"619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6\": rpc error: code = NotFound desc = could not find container \"619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6\": container with ID starting with 619e440be3a471793b0edfd099838812e99f391408d0b9eb0e43051372d6c0e6 not found: ID does not exist" Oct 30 22:11:04.627510 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.627447 5508 scope.go:122] "RemoveContainer" containerID="0954ee3127d8945a33df82e25d65e9edcb8d41babcdfa55153bff38ffe3a4920" Oct 30 22:11:04.638573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.638555 5508 scope.go:122] "RemoveContainer" containerID="e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543" Oct 30 22:11:04.649215 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.649194 5508 scope.go:122] "RemoveContainer" containerID="e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543" Oct 30 22:11:04.649561 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:04.649536 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543\": container with ID starting with e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543 not found: ID does not exist" containerID="e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543" Oct 30 22:11:04.649657 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.649568 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543"} err="failed to get container status \"e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543\": rpc error: code = NotFound desc = could not find container \"e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543\": container with ID starting with e69735840d53ae238958b5e33062e740ca2364ae4b30e35d65d7e6f1c8e51543 not found: ID does not exist" Oct 30 22:11:04.773083 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.773001 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:04.781814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.781785 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7"] Oct 30 22:11:04.786649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.786626 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qn5r7"] Oct 30 22:11:04.909245 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.909210 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv"] Oct 30 22:11:04.913424 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.913030 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-wlgdv"] Oct 30 22:11:04.996833 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:04.996793 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.025465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.025378 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx"] Oct 30 22:11:05.026889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.026851 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-wkhmx"] Oct 30 22:11:05.189980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.189930 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.632253 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.632214 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" event={"ID":"323cc524-025b-4d6d-bf45-845ac6156590","Type":"ContainerStarted","Data":"892dbd9302762b49f303936b27f53144b060f158ddfd46ffdc2330828859d54b"} Oct 30 22:11:05.761902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.761847 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podStartSLOduration=2.761826003 podStartE2EDuration="2.761826003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:05.748201501 +0000 UTC m=+1675.937739198" watchObservedRunningTime="2026-10-30 22:11:05.761826003 +0000 UTC m=+1675.951363679" Oct 30 22:11:05.869961 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.869922 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.955385 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.955284 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" path="/var/lib/kubelet/pods/03ae515e-2442-4885-b4d9-d1d54a2d8eda/volumes" Oct 30 22:11:05.955936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.955913 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" path="/var/lib/kubelet/pods/03ecca1a-530c-4f48-9e42-a7d7408ba81f/volumes" Oct 30 22:11:05.956481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.956462 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0527abff-b56e-457a-a62f-cae88f284eea" path="/var/lib/kubelet/pods/0527abff-b56e-457a-a62f-cae88f284eea/volumes" Oct 30 22:11:05.957263 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:05.957241 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80380e39-4926-405e-a689-9aea39dd12ec" path="/var/lib/kubelet/pods/80380e39-4926-405e-a689-9aea39dd12ec/volumes" Oct 30 22:11:06.013865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:06.013832 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:06.019370 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:06.019344 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:06.640779 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:06.640748 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:06.641692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:06.641674 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:11:07.035851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:07.035759 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:07.280594 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:07.280559 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:08.663161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:08.663130 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:08.898937 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:08.898904 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:09.088870 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:09.088833 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.374980 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:10.374945 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:11:10.620338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:10.620289 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.822414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:10.821586 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:11.036029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:11.036001 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.013568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:13.013532 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.245352 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:13.245283 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.369328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:13.367002 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz"] Oct 30 22:11:13.369328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:13.367327 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" containerID="cri-o://3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807" gracePeriod=30 Oct 30 22:11:13.519931 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:13.517888 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:14.066855 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:14.066816 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:14.372269 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:14.372229 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.640061 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:16.640010 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:18.680738 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.680677 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:11:18.793432 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793397 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793534 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793564 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793634 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793663 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793699 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.793729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793707 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793807 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.793979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793904 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.793979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793935 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:18.793979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.793941 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.794186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794031 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token\") pod \"e37a9dba-9477-4e36-9e10-422bf6286e1b\" (UID: \"e37a9dba-9477-4e36-9e10-422bf6286e1b\") " Oct 30 22:11:18.794186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794103 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.794327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794212 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.794938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794911 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.795054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794945 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37a9dba-9477-4e36-9e10-422bf6286e1b-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.795054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794959 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.795054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794974 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.795054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.794987 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.796481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.796453 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.796583 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.796499 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "kube-api-access-zmd89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:18.796653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.796598 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:18.796726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.796707 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token" pod "e37a9dba-9477-4e36-9e10-422bf6286e1b" (UID: "e37a9dba-9477-4e36-9e10-422bf6286e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:18.825379 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.825353 5508 generic.go:356] "Generic (PLEG): container finished" podID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerID="3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807" exitCode=0 Oct 30 22:11:18.825471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.825417 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" Oct 30 22:11:18.825512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.825420 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" event={"ID":"e37a9dba-9477-4e36-9e10-422bf6286e1b","Type":"ContainerDied","Data":"3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807"} Oct 30 22:11:18.825585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.825557 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz" event={"ID":"e37a9dba-9477-4e36-9e10-422bf6286e1b","Type":"ContainerDied","Data":"e17ea7b1d0c5fcee461053a82e4f4a6b9401fd1eb96d85c7003163b3c751384c"} Oct 30 22:11:18.825585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.825580 5508 scope.go:122] "RemoveContainer" containerID="3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807" Oct 30 22:11:18.835906 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.835888 5508 scope.go:122] "RemoveContainer" containerID="3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807" Oct 30 22:11:18.836186 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:18.836157 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807\": container with ID starting with 3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807 not found: ID does not exist" containerID="3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807" Oct 30 22:11:18.836237 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.836186 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807"} err="failed to get container status \"3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807\": rpc error: code = NotFound desc = could not find container \"3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807\": container with ID starting with 3b8231d091b100055fa56c7dbcc463eb74f5459a06c5b647db5dd33ac5478807 not found: ID does not exist" Oct 30 22:11:18.896377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.896348 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmd89\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-kube-api-access-zmd89\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.896377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.896373 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.896585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.896386 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.896585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.896394 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37a9dba-9477-4e36-9e10-422bf6286e1b-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.972526 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.972442 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz"] Oct 30 22:11:18.978528 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:18.978496 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-qstgz"] Oct 30 22:11:19.953634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:19.953606 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" path="/var/lib/kubelet/pods/e37a9dba-9477-4e36-9e10-422bf6286e1b/volumes" Oct 30 22:11:21.277117 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.277088 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:21.499012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.498973 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:21.861240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.861208 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484"] Oct 30 22:11:21.862878 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.862856 5508 cpu_manager.go:414] "containerMap: removing container" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" Oct 30 22:11:21.863015 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.863005 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" Oct 30 22:11:21.863111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.863103 5508 cpu_manager.go:414] "containerMap: removing container" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" Oct 30 22:11:21.863164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.863157 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" Oct 30 22:11:21.863236 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.863230 5508 cpu_manager.go:414] "containerMap: removing container" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" Oct 30 22:11:21.863315 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.863291 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" Oct 30 22:11:21.866439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.866417 5508 cpu_manager.go:414] "containerMap: removing container" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" Oct 30 22:11:21.866552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.866543 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" Oct 30 22:11:21.866879 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.866866 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37a9dba-9477-4e36-9e10-422bf6286e1b" containerName="istio-proxy" Oct 30 22:11:21.866961 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.866954 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0527abff-b56e-457a-a62f-cae88f284eea" containerName="istio-proxy" Oct 30 22:11:21.867031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.867024 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ecca1a-530c-4f48-9e42-a7d7408ba81f" containerName="istio-proxy" Oct 30 22:11:21.867091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.867085 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ae515e-2442-4885-b4d9-d1d54a2d8eda" containerName="istio-proxy" Oct 30 22:11:21.897611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.897571 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:21.898419 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.897298 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484"] Oct 30 22:11:21.950757 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:21.950720 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-e431fc\"" Oct 30 22:11:22.006679 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.006630 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-e431fc-dockercfg-qfm46\"" Oct 30 22:11:22.023218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.023170 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czr9v\" (UniqueName: \"kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v\") pod \"payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.037533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.023851 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.111990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.111904 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.125484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.125442 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.125655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.125598 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czr9v\" (UniqueName: \"kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v\") pod \"payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.126137 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.126117 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.183772 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.183735 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czr9v\" (UniqueName: \"kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v\") pod \"payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.219861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.219827 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.613890 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.613859 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484"] Oct 30 22:11:22.614950 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:11:22.614925 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5d5ef8_df27_4f0a_bb33_2a64f409480a.slice/crio-16da40ad8a752442817e3d919ac93df49cb9eaa5a1008e11261fa4034d47a68d WatchSource:0}: Error finding container 16da40ad8a752442817e3d919ac93df49cb9eaa5a1008e11261fa4034d47a68d: Status 404 returned error can't find the container with id 16da40ad8a752442817e3d919ac93df49cb9eaa5a1008e11261fa4034d47a68d Oct 30 22:11:22.885518 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.885382 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" event={"ID":"bf5d5ef8-df27-4f0a-bb33-2a64f409480a","Type":"ContainerStarted","Data":"1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea"} Oct 30 22:11:22.885518 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.885488 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:22.885730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:22.885514 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" event={"ID":"bf5d5ef8-df27-4f0a-bb33-2a64f409480a","Type":"ContainerStarted","Data":"16da40ad8a752442817e3d919ac93df49cb9eaa5a1008e11261fa4034d47a68d"} Oct 30 22:11:23.006256 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:23.006200 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" podStartSLOduration=2.006183116 podStartE2EDuration="2.006183116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:23.001879381 +0000 UTC m=+1693.191417059" watchObservedRunningTime="2026-10-30 22:11:23.006183116 +0000 UTC m=+1693.195720792" Oct 30 22:11:26.940660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:26.940629 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn"] Oct 30 22:11:26.945039 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:26.945009 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.000672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.000640 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.000837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.000765 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.000893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.000864 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.001018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.000996 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vsv\" (UniqueName: \"kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.017241 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.017211 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8f1d78db-serving-cert\"" Oct 30 22:11:27.052798 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.052766 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn"] Oct 30 22:11:27.102355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.102259 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.102511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.102441 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.102589 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:27.102562 5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-serving-cert: secret "maas-api-e2e-mt-8f1d78db-serving-cert" not found Oct 30 22:11:27.102659 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.102585 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.102659 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:27.102635 5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls podName:cf7694c5-5785-42c3-90a8-858c708f7904 nodeName:}" failed. No retries permitted until 2026-10-30 22:11:27.602613332 +0000 UTC m=+1697.792150984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls") pod "maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" (UID: "cf7694c5-5785-42c3-90a8-858c708f7904") : secret "maas-api-e2e-mt-8f1d78db-serving-cert" not found Oct 30 22:11:27.102773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.102698 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vsv\" (UniqueName: \"kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.103326 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.103280 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.105331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.105294 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.190348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.189781 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vsv\" (UniqueName: \"kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.216441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.212703 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:27.396843 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.396527 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:27.612738 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.612699 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.615402 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.615378 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\") pod \"maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:27.858722 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:27.858688 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:28.239580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:28.239548 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn"] Oct 30 22:11:28.241212 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:11:28.241187 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7694c5_5785_42c3_90a8_858c708f7904.slice/crio-ba5b2bd8b6c70a5056d514daacf5c6320ab0dea78f0d3b12fee393b8f7c9bbfc WatchSource:0}: Error finding container ba5b2bd8b6c70a5056d514daacf5c6320ab0dea78f0d3b12fee393b8f7c9bbfc: Status 404 returned error can't find the container with id ba5b2bd8b6c70a5056d514daacf5c6320ab0dea78f0d3b12fee393b8f7c9bbfc Oct 30 22:11:28.242677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:28.242658 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:11:28.928564 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:28.928522 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" event={"ID":"cf7694c5-5785-42c3-90a8-858c708f7904","Type":"ContainerStarted","Data":"ba5b2bd8b6c70a5056d514daacf5c6320ab0dea78f0d3b12fee393b8f7c9bbfc"} Oct 30 22:11:29.936744 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:29.936701 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" event={"ID":"cf7694c5-5785-42c3-90a8-858c708f7904","Type":"ContainerStarted","Data":"abc5aa4c706123a25646965d153801e164d0807c5e8c4bb4914d317d0a9a7403"} Oct 30 22:11:29.937196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:29.936836 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:30.068464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:30.068059 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" podStartSLOduration=3.252704969 podStartE2EDuration="4.06803247s" totalImagesPullingTime="815.327501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:30.061777006 +0000 UTC m=+1700.251314708" watchObservedRunningTime="2026-10-30 22:11:30.06803247 +0000 UTC m=+1700.257570182" Oct 30 22:11:30.730245 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:30.730198 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:32.385674 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:32.374988 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:33.894894 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:33.894865 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:11:35.953935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:35.953907 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:11:37.394377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:37.394278 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:40.373434 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:11:40.373392 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:11:42.406790 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:42.406742 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:47.448098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:47.448021 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.283620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:53.283521 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:57.168082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:57.168039 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:57.472518 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:57.472424 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.391417 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.391374 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4"] Oct 30 22:11:58.391829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.391628 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" containerID="cri-o://b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee" gracePeriod=30 Oct 30 22:11:58.439825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.439790 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc"] Oct 30 22:11:58.440042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.440006 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" containerID="cri-o://a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f" gracePeriod=30 Oct 30 22:11:58.443135 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.443108 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29"] Oct 30 22:11:58.443361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.443317 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" containerID="cri-o://3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918" gracePeriod=30 Oct 30 22:11:58.468085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.468056 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.489088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.489059 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v"] Oct 30 22:11:58.489286 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.489267 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" containerID="cri-o://28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961" gracePeriod=30 Oct 30 22:11:58.819560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:58.819530 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:59.157728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:59.157647 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:59.692079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:11:59.692044 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:01.248595 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:01.248561 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:12:01.248987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:01.248625 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:12:02.192465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:02.192417 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:12:02.192644 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:02.192502 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:12:02.364542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:02.364500 5508 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:12:02.364920 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:02.364576 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:12:03.743830 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.743795 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:12:03.812182 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.812062 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 22:12:03.869292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.869263 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:12:03.877285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.877263 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 22:12:03.886034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886008 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886152 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886071 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886163 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886270 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886206 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886270 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886253 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886350 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886454 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.886555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886533 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.887122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886678 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy\") pod \"37368dad-c429-479b-a803-4d4429f6bb44\" (UID: \"37368dad-c429-479b-a803-4d4429f6bb44\") " Oct 30 22:12:03.887122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886602 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:03.887122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886869 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.887122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.886881 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.887122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.887116 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.887572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.887544 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37368dad-c429-479b-a803-4d4429f6bb44-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.887645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.887572 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.887645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.887587 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.887645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.887602 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.887804 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.887725 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.888671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.888646 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "kube-api-access-tskrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.888793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.888770 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.888900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.888880 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.889238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.889212 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo" pod "37368dad-c429-479b-a803-4d4429f6bb44" (UID: "37368dad-c429-479b-a803-4d4429f6bb44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:03.988020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.987992 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988061 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988117 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988177 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.988250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988221 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988248 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988318 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.988380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988359 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988384 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988427 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988440 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988473 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988520 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988561 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988585 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988619 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988654 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988710 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988738 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988791 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988832 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988862 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988917 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988945 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.988903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.988949 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989065 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket\") pod \"d73eb82d-1732-4b29-b53a-104b03d75e32\" (UID: \"d73eb82d-1732-4b29-b53a-104b03d75e32\") " Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989258 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989342 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6\") pod \"7f010838-ed32-44bc-aa3d-f59b6400be81\" (UID: \"7f010838-ed32-44bc-aa3d-f59b6400be81\") " Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989405 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989437 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket\") pod \"47652914-4a96-45a0-948f-5a2d262e98b5\" (UID: \"47652914-4a96-45a0-948f-5a2d262e98b5\") " Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989448 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989697 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989720 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989925 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.989939 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990188 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990234 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990455 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990469 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990483 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990493 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990509 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990523 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990539 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990553 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f010838-ed32-44bc-aa3d-f59b6400be81-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990568 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskrc\" (UniqueName: \"kubernetes.io/projected/37368dad-c429-479b-a803-4d4429f6bb44-kube-api-access-tskrc\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990585 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990599 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37368dad-c429-479b-a803-4d4429f6bb44-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990614 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990628 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990641 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990650 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990655 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37368dad-c429-479b-a803-4d4429f6bb44-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.990703 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.991041 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.991153 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "kube-api-access-4m9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.991697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.991206 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.993733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.993054 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "kube-api-access-dzxf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.993905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.993837 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.994041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.993956 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.994041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.994004 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "kube-api-access-tpflv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.994041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.994014 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo" pod "47652914-4a96-45a0-948f-5a2d262e98b5" (UID: "47652914-4a96-45a0-948f-5a2d262e98b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:03.994223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.994070 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:03.994223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.994084 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.994223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.994181 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy" pod "d73eb82d-1732-4b29-b53a-104b03d75e32" (UID: "d73eb82d-1732-4b29-b53a-104b03d75e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.994624 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.994587 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:12:03.995563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.995544 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:03.995886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:03.995862 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo" pod "7f010838-ed32-44bc-aa3d-f59b6400be81" (UID: "7f010838-ed32-44bc-aa3d-f59b6400be81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092367 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092401 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxf6\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-kube-api-access-dzxf6\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092418 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092431 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092445 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092461 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092475 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpflv\" (UniqueName: \"kubernetes.io/projected/47652914-4a96-45a0-948f-5a2d262e98b5-kube-api-access-tpflv\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092490 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092505 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092518 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47652914-4a96-45a0-948f-5a2d262e98b5-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092534 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9vt\" (UniqueName: \"kubernetes.io/projected/d73eb82d-1732-4b29-b53a-104b03d75e32-kube-api-access-4m9vt\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092550 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73eb82d-1732-4b29-b53a-104b03d75e32-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092564 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73eb82d-1732-4b29-b53a-104b03d75e32-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092578 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47652914-4a96-45a0-948f-5a2d262e98b5-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092592 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47652914-4a96-45a0-948f-5a2d262e98b5-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092605 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092617 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.092655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.092629 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f010838-ed32-44bc-aa3d-f59b6400be81-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:04.187762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.187726 5508 generic.go:356] "Generic (PLEG): container finished" podID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerID="a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f" exitCode=0 Oct 30 22:12:04.187950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.187795 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" Oct 30 22:12:04.187950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.187810 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" event={"ID":"d73eb82d-1732-4b29-b53a-104b03d75e32","Type":"ContainerDied","Data":"a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f"} Oct 30 22:12:04.187950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.187878 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc" event={"ID":"d73eb82d-1732-4b29-b53a-104b03d75e32","Type":"ContainerDied","Data":"cebe6d6d9a76a11646a86c89884253b27381c672a6281c23fc52e5596c844b40"} Oct 30 22:12:04.187950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.187908 5508 scope.go:122] "RemoveContainer" containerID="a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f" Oct 30 22:12:04.190065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.190038 5508 generic.go:356] "Generic (PLEG): container finished" podID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerID="3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918" exitCode=0 Oct 30 22:12:04.190396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.190101 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" Oct 30 22:12:04.190396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.190127 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" event={"ID":"7f010838-ed32-44bc-aa3d-f59b6400be81","Type":"ContainerDied","Data":"3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918"} Oct 30 22:12:04.190396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.190174 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29" event={"ID":"7f010838-ed32-44bc-aa3d-f59b6400be81","Type":"ContainerDied","Data":"dfe92aee550bb4370a36d7daef9a04a1862740e6202499ffb89ecf562085c019"} Oct 30 22:12:04.191945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.191923 5508 generic.go:356] "Generic (PLEG): container finished" podID="47652914-4a96-45a0-948f-5a2d262e98b5" containerID="28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961" exitCode=0 Oct 30 22:12:04.194362 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.192067 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" event={"ID":"47652914-4a96-45a0-948f-5a2d262e98b5","Type":"ContainerDied","Data":"28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961"} Oct 30 22:12:04.194362 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.192096 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" Oct 30 22:12:04.194362 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.192113 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v" event={"ID":"47652914-4a96-45a0-948f-5a2d262e98b5","Type":"ContainerDied","Data":"8a670961127f0ded46b1d10ec5375ad8198a7bc61290c950e78f2dbb699bc910"} Oct 30 22:12:04.202824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.201563 5508 scope.go:122] "RemoveContainer" containerID="559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85" Oct 30 22:12:04.202824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.201821 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" Oct 30 22:12:04.202824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.201869 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" event={"ID":"37368dad-c429-479b-a803-4d4429f6bb44","Type":"ContainerDied","Data":"b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee"} Oct 30 22:12:04.202824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.202608 5508 generic.go:356] "Generic (PLEG): container finished" podID="37368dad-c429-479b-a803-4d4429f6bb44" containerID="b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee" exitCode=0 Oct 30 22:12:04.202824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.202643 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4" event={"ID":"37368dad-c429-479b-a803-4d4429f6bb44","Type":"ContainerDied","Data":"cdc807d4b092ae536239be024ae949e87c371279251ce87266668f1c0823313a"} Oct 30 22:12:04.215822 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.215798 5508 scope.go:122] "RemoveContainer" containerID="a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f" Oct 30 22:12:04.216156 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:04.216126 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f\": container with ID starting with a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f not found: ID does not exist" containerID="a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f" Oct 30 22:12:04.216275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.216164 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f"} err="failed to get container status \"a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f\": rpc error: code = NotFound desc = could not find container \"a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f\": container with ID starting with a8b2fb390d34bc262d2701670f0ba91c0052f02197bde552d9146d3fea55b19f not found: ID does not exist" Oct 30 22:12:04.216275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.216185 5508 scope.go:122] "RemoveContainer" containerID="559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85" Oct 30 22:12:04.216503 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:04.216481 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85\": container with ID starting with 559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85 not found: ID does not exist" containerID="559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85" Oct 30 22:12:04.216565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.216508 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85"} err="failed to get container status \"559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85\": rpc error: code = NotFound desc = could not find container \"559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85\": container with ID starting with 559a119973fed0d483a554cd777b5e8e1adee211c82e4b9d39e0a4133cb33a85 not found: ID does not exist" Oct 30 22:12:04.216565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.216524 5508 scope.go:122] "RemoveContainer" containerID="3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918" Oct 30 22:12:04.228210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.228189 5508 scope.go:122] "RemoveContainer" containerID="3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918" Oct 30 22:12:04.228645 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:04.228621 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918\": container with ID starting with 3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918 not found: ID does not exist" containerID="3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918" Oct 30 22:12:04.228736 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.228653 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918"} err="failed to get container status \"3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918\": rpc error: code = NotFound desc = could not find container \"3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918\": container with ID starting with 3a1931f5796840a6cb9c730b6480d80e5f05591123fad8d68046f13797ef1918 not found: ID does not exist" Oct 30 22:12:04.228736 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.228670 5508 scope.go:122] "RemoveContainer" containerID="28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961" Oct 30 22:12:04.241088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.241066 5508 scope.go:122] "RemoveContainer" containerID="28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961" Oct 30 22:12:04.241581 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:04.241430 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961\": container with ID starting with 28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961 not found: ID does not exist" containerID="28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961" Oct 30 22:12:04.241581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.241471 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961"} err="failed to get container status \"28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961\": rpc error: code = NotFound desc = could not find container \"28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961\": container with ID starting with 28d847078aa3481e886b91b0230a2e443b0de380b7f364162100d898c10a3961 not found: ID does not exist" Oct 30 22:12:04.241581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.241495 5508 scope.go:122] "RemoveContainer" containerID="b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee" Oct 30 22:12:04.252541 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.252517 5508 scope.go:122] "RemoveContainer" containerID="b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee" Oct 30 22:12:04.252859 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:04.252838 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee\": container with ID starting with b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee not found: ID does not exist" containerID="b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee" Oct 30 22:12:04.252942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.252863 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee"} err="failed to get container status \"b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee\": rpc error: code = NotFound desc = could not find container \"b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee\": container with ID starting with b815647cfc217d69402ce10b1ab4027b5d0df2a4df7eb21f403c17d99ae359ee not found: ID does not exist" Oct 30 22:12:04.426281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.426159 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4"] Oct 30 22:12:04.430599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.430572 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-n5kg4"] Oct 30 22:12:04.543950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.543853 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v"] Oct 30 22:12:04.553715 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.553676 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-lxc9v"] Oct 30 22:12:04.593326 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.588752 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:04.664657 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.664626 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29"] Oct 30 22:12:04.670300 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.670274 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-qwm29"] Oct 30 22:12:04.784042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.784008 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc"] Oct 30 22:12:04.800264 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:04.800186 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-fjbfc"] Oct 30 22:12:05.896808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:05.896771 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:05.956940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:05.956906 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37368dad-c429-479b-a803-4d4429f6bb44" path="/var/lib/kubelet/pods/37368dad-c429-479b-a803-4d4429f6bb44/volumes" Oct 30 22:12:05.957524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:05.957509 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" path="/var/lib/kubelet/pods/47652914-4a96-45a0-948f-5a2d262e98b5/volumes" Oct 30 22:12:05.957962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:05.957948 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" path="/var/lib/kubelet/pods/7f010838-ed32-44bc-aa3d-f59b6400be81/volumes" Oct 30 22:12:05.958737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:05.958722 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" path="/var/lib/kubelet/pods/d73eb82d-1732-4b29-b53a-104b03d75e32/volumes" Oct 30 22:12:10.373468 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:10.373431 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:12:11.804702 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:11.804660 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:12.277506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:12.277470 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:17.314342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:17.311417 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:17.770842 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:17.770800 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:22.707008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:22.706971 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:22.951434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:22.951394 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.234541 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:23.234493 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.431763 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:23.431717 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:27.888524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:27.888488 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:28.132573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:28.132537 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:28.541327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:28.540857 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:31.143737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:31.143683 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:33.104820 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:33.104773 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:38.186990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:38.186956 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:40.201859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:40.201826 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:40.374235 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:12:40.374203 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:12:40.663081 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:40.663049 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:41.333433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:41.333385 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:41.710453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:41.710355 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:43.740962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:43.740900 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.025586 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:46.025556 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.264266 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:46.264230 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.458008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:46.457971 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.177342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.177266 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.429376 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.429270 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.684199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.684106 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.982425 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.982353 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x"] Oct 30 22:12:49.983368 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983342 5508 cpu_manager.go:414] "containerMap: removing container" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" Oct 30 22:12:49.983368 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983365 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983380 5508 cpu_manager.go:414] "containerMap: removing container" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983387 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983423 5508 cpu_manager.go:414] "containerMap: removing container" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983430 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983473 5508 cpu_manager.go:414] "containerMap: removing container" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983480 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983493 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" Oct 30 22:12:49.983603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983500 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" Oct 30 22:12:49.983977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983670 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" Oct 30 22:12:49.983977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983690 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="47652914-4a96-45a0-948f-5a2d262e98b5" containerName="istio-proxy" Oct 30 22:12:49.983977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983701 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f010838-ed32-44bc-aa3d-f59b6400be81" containerName="istio-proxy" Oct 30 22:12:49.983977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983714 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73eb82d-1732-4b29-b53a-104b03d75e32" containerName="istio-proxy" Oct 30 22:12:49.983977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.983723 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="37368dad-c429-479b-a803-4d4429f6bb44" containerName="istio-proxy" Oct 30 22:12:49.987787 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:49.987761 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.036703 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.035769 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x"] Oct 30 22:12:50.080917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.080881 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-0a7679-dockercfg-cs4br\"" Oct 30 22:12:50.081105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.081045 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0a7679\"" Oct 30 22:12:50.147615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.147555 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vdc8\" (UniqueName: \"kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8\") pod \"payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.147818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.147687 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.263386 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.249588 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vdc8\" (UniqueName: \"kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8\") pod \"payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.264149 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.264117 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.289672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.289639 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.353327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.352976 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vdc8\" (UniqueName: \"kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8\") pod \"payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:50.580019 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.579982 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:50.615716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:50.613646 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:51.440678 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:51.440639 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x"] Oct 30 22:12:51.701364 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:51.701208 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" event={"ID":"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d","Type":"ContainerStarted","Data":"565276c48730636d66967397bfb6da18d8ae1d049b43e210401e3dcb78792178"} Oct 30 22:12:52.784374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.777786 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" event={"ID":"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d","Type":"ContainerStarted","Data":"ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12"} Oct 30 22:12:52.784374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.778887 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:12:52.896338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.894398 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:52.896338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.895874 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:52.934337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.923413 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" podStartSLOduration=3.9233923649999998 podStartE2EDuration="3.923392365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:52.914344076 +0000 UTC m=+1783.103881780" watchObservedRunningTime="2026-10-30 22:12:52.923392365 +0000 UTC m=+1783.112930043" Oct 30 22:12:52.938494 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.938452 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:52.976420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:52.976378 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:12:53.031231 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.030428 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:53.106340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.105647 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:53.824276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.823463 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:53.903457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.903414 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:53.904682 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.904659 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:12:53.942885 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.942847 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" podUID="b1b6787b-0e4c-4564-a9f0-5e84ea279f38" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:53.946933 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:53.946900 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:12:54.005790 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.005745 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 22:12:54.033185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.032634 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:54.058342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.040190 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:12:54.058685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.058664 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:12:54.107519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.107401 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:54.164331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.133133 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:12:54.472952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.467532 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:54.799382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.796934 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:12:54.831340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.830560 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:54.831340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.831109 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:12:54.920197 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.920158 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:12:54.921539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.921518 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:12:54.925644 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.925624 5508 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:12:54.965333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:54.947758 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r" Oct 30 22:12:55.016329 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:55.012517 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:12:55.068778 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:55.062140 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:12:55.094909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:55.094878 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:12:55.131923 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:55.131895 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:12:56.119485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:12:56.119442 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:00.432975 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:00.432937 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:01.777350 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.777293 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:01.965670 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.965640 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:01.983786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.983748 5508 generic.go:356] "Generic (PLEG): container finished" podID="323cc524-025b-4d6d-bf45-845ac6156590" containerID="892dbd9302762b49f303936b27f53144b060f158ddfd46ffdc2330828859d54b" exitCode=0 Oct 30 22:13:01.983960 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.983819 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" event={"ID":"323cc524-025b-4d6d-bf45-845ac6156590","Type":"ContainerDied","Data":"892dbd9302762b49f303936b27f53144b060f158ddfd46ffdc2330828859d54b"} Oct 30 22:13:01.984338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.984300 5508 scope.go:122] "RemoveContainer" containerID="892dbd9302762b49f303936b27f53144b060f158ddfd46ffdc2330828859d54b" Oct 30 22:13:01.985993 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.985964 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:01.986068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.986036 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:01.986113 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:01.986069 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:02.487520 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:02.487480 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:03.006927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:03.006891 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" event={"ID":"323cc524-025b-4d6d-bf45-845ac6156590","Type":"ContainerStarted","Data":"a5175a9b856c3d6cb6e43ee5ea9c4a8ca7040407fe82038cfd43164631c2eb20"} Oct 30 22:13:03.013503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:03.013479 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:13:03.014908 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:03.014882 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:13:03.015006 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:03.014927 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:13:04.013142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:04.013109 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:13:04.013608 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:04.013170 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:13:04.929720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:04.929693 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:13:05.014116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:05.014082 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:13:05.014630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:05.014151 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:13:06.013500 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:06.013467 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:13:06.013711 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:06.013540 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:13:06.641457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:06.641415 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:13:07.014156 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:07.014077 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:13:07.014320 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:07.014154 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:13:07.517330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:07.516855 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:08.013428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:08.013390 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:13:08.013923 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:08.013460 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:13:09.018728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:09.018699 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:13:09.019935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:09.019914 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:13:09.063681 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:09.063642 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:10.373988 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:13:10.373953 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:13:13.114922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:13.114888 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:19.565342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:19.564166 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:24.644669 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:24.644624 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:29.452774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:29.452729 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:31.469348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:31.469287 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:34.849236 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:34.849202 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:40.130342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:40.129788 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:40.373195 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:13:40.373160 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:13:44.017903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:44.017870 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:44.416872 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:44.416840 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:44.566692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:44.566659 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9"] Oct 30 22:13:44.566886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:44.566865 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" containerID="cri-o://3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7" gracePeriod=30 Oct 30 22:13:45.284352 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.284320 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2"] Oct 30 22:13:45.284826 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.284571 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" containerID="cri-o://093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69" gracePeriod=30 Oct 30 22:13:45.334681 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.334637 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9"] Oct 30 22:13:45.338412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.338378 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.394554 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.394529 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-db492e-openshift-default-dockercfg-b5s7t\"" Oct 30 22:13:45.395256 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395237 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395268 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395393 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395487 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395535 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395518 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6lv\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395541 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395599 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395725 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395695 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.395810 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.395795 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.406220 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.406193 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9"] Oct 30 22:13:45.442716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.442669 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:45.499014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.498629 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.499631 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.499293 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.499826 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.499804 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.500088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.500068 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.500220 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.500199 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.500439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.500417 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.500601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.500583 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6lv\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.500767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.500739 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.502646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.502615 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.502896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.502877 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.504713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.504685 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.504815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.500654 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.505428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.505393 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.505682 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.505657 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.506474 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.506453 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.509120 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.509100 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.586124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.586090 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6lv\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.591395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.589268 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token\") pod \"e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.654994 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.654960 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:45.731221 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:45.731184 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:46.090429 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:13:46.090400 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676bfdef_67cf_43fb_8c33_2b6a06f6c4f9.slice/crio-bbbe8df00f3dbf09f7743cfe086e397385fed812f752bcbe8380256d3e61ba25 WatchSource:0}: Error finding container bbbe8df00f3dbf09f7743cfe086e397385fed812f752bcbe8380256d3e61ba25: Status 404 returned error can't find the container with id bbbe8df00f3dbf09f7743cfe086e397385fed812f752bcbe8380256d3e61ba25 Oct 30 22:13:46.092998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.092970 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9"] Oct 30 22:13:46.093240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.093212 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:46.093281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.093273 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:46.093357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.093343 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:13:46.167344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.167274 5508 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:46.167471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.167374 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:46.331074 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.331023 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" event={"ID":"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9","Type":"ContainerStarted","Data":"dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976"} Oct 30 22:13:46.331559 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.331092 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" event={"ID":"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9","Type":"ContainerStarted","Data":"bbbe8df00f3dbf09f7743cfe086e397385fed812f752bcbe8380256d3e61ba25"} Oct 30 22:13:46.453014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.452803 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podStartSLOduration=1.45278587 podStartE2EDuration="1.45278587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:46.450956873 +0000 UTC m=+1836.640494550" watchObservedRunningTime="2026-10-30 22:13:46.45278587 +0000 UTC m=+1836.642323539" Oct 30 22:13:46.655555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:46.655520 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:47.599581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:47.599542 5508 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:13:47.599972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:47.599610 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:13:47.661957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:47.661925 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:48.345621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:48.345584 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:48.346514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:48.346492 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:13:49.338066 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.338034 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.541842 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.541800 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.715241 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.715205 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.848254 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.848202 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 22:13:49.866890 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.866860 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.866998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.866978 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867037 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867013 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867109 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867187 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867173 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867243 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867205 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867297 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867239 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867317 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867335 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:49.867487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867407 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data\") pod \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\" (UID: \"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd\") " Oct 30 22:13:49.867487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867425 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:49.867889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867693 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:49.867889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867767 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:49.867889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.867862 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:49.868152 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.868133 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.868214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.868158 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.868214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.868175 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.868214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.868189 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.868214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.868202 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.869736 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.869717 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:49.869795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.869733 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:49.869929 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.869900 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:49.869929 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.869920 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn" pod "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" (UID: "7c8d7813-6b35-45ff-ac01-d16abdbbf2fd"). InnerVolumeSpecName "kube-api-access-lmzzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:49.969480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.969448 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.969626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.969482 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.969626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.969499 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:49.969626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:49.969516 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmzzn\" (UniqueName: \"kubernetes.io/projected/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd-kube-api-access-lmzzn\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.360249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.360209 5508 generic.go:356] "Generic (PLEG): container finished" podID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerID="3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7" exitCode=0 Oct 30 22:13:50.360249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.360259 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" Oct 30 22:13:50.360799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.360292 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" event={"ID":"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd","Type":"ContainerDied","Data":"3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7"} Oct 30 22:13:50.360799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.360420 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9" event={"ID":"7c8d7813-6b35-45ff-ac01-d16abdbbf2fd","Type":"ContainerDied","Data":"45372ea0653077fe6659a1225727b9385d9c1c6229e94f71c4158297ff799021"} Oct 30 22:13:50.360799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.360451 5508 scope.go:122] "RemoveContainer" containerID="3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7" Oct 30 22:13:50.399180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.399154 5508 scope.go:122] "RemoveContainer" containerID="3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7" Oct 30 22:13:50.399656 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:13:50.399612 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7\": container with ID starting with 3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7 not found: ID does not exist" containerID="3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7" Oct 30 22:13:50.399785 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.399668 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7"} err="failed to get container status \"3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7\": rpc error: code = NotFound desc = could not find container \"3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7\": container with ID starting with 3059dd0904e8cf0d2b5ff96f752664297879ec65aa21709459162170f12a0ff7 not found: ID does not exist" Oct 30 22:13:50.532030 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.531994 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9"] Oct 30 22:13:50.572881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.572368 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:13:50.575777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.575752 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nxcn9"] Oct 30 22:13:50.689530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.689445 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.689791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.689535 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.689791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.689642 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.689791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.689685 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.689791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.689750 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.690027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.689892 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.690606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.690584 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.690842 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.690823 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.691026 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.690972 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.691502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.690639 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.691502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.690799 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.691502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.691410 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket\") pod \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\" (UID: \"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c\") " Oct 30 22:13:50.691704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.691596 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:50.693275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.692796 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.693275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.692900 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.693275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.692920 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.693275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.692935 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.693275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.692949 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.694137 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.694097 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "kube-api-access-scqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:50.694702 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.694655 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:50.695025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.695003 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:50.695295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.695261 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy" pod "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" (UID: "8677b9d5-ec7b-4b07-bfdb-585af3c3f40c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:50.794393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.794364 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.794393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.794390 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scqlw\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-kube-api-access-scqlw\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.794393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.794401 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.794617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.794409 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:50.794617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:50.794418 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.116871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.116815 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.367775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.367690 5508 generic.go:356] "Generic (PLEG): container finished" podID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerID="093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69" exitCode=0 Oct 30 22:13:51.367775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.367738 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" event={"ID":"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c","Type":"ContainerDied","Data":"093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69"} Oct 30 22:13:51.368294 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.367797 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" event={"ID":"8677b9d5-ec7b-4b07-bfdb-585af3c3f40c","Type":"ContainerDied","Data":"bbfe48bf2cf164be3d33a84cf9a4cb1706873fbc1af0ba4cb5a5b6e41d77b24b"} Oct 30 22:13:51.368294 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.367756 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2" Oct 30 22:13:51.368294 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.367820 5508 scope.go:122] "RemoveContainer" containerID="093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69" Oct 30 22:13:51.379184 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.379167 5508 scope.go:122] "RemoveContainer" containerID="093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69" Oct 30 22:13:51.379465 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:13:51.379442 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69\": container with ID starting with 093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69 not found: ID does not exist" containerID="093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69" Oct 30 22:13:51.379527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.379473 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69"} err="failed to get container status \"093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69\": rpc error: code = NotFound desc = could not find container \"093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69\": container with ID starting with 093d9ff167abda9c6880e7450b614eec53b38e18570848a7e559b6dc006f9f69 not found: ID does not exist" Oct 30 22:13:51.534395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.534363 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2"] Oct 30 22:13:51.584082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.584045 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-8szp2"] Oct 30 22:13:51.955232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.955197 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" path="/var/lib/kubelet/pods/7c8d7813-6b35-45ff-ac01-d16abdbbf2fd/volumes" Oct 30 22:13:51.955913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:51.955896 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" path="/var/lib/kubelet/pods/8677b9d5-ec7b-4b07-bfdb-585af3c3f40c/volumes" Oct 30 22:13:53.851814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:53.851775 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.455981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.455949 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.793692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.793605 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.871106 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.871071 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7"] Oct 30 22:13:56.871886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.871860 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" Oct 30 22:13:56.871886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.871885 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" Oct 30 22:13:56.872124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.871912 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" Oct 30 22:13:56.872124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.871918 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" Oct 30 22:13:56.872124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.872091 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8677b9d5-ec7b-4b07-bfdb-585af3c3f40c" containerName="istio-proxy" Oct 30 22:13:56.872124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.872116 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8d7813-6b35-45ff-ac01-d16abdbbf2fd" containerName="istio-proxy" Oct 30 22:13:56.878277 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.878215 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:56.933709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.933685 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-db492e\"" Oct 30 22:13:56.991874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:56.991833 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-db492e-dockercfg-g2vvd\"" Oct 30 22:13:57.007810 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.007774 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7"] Oct 30 22:13:57.102401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.087358 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:57.103245 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.089479 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28"] Oct 30 22:13:57.114449 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.109111 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.126672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.126643 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.127127 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.127103 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6tp\" (UniqueName: \"kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp\") pod \"payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.192912 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.192825 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28"] Oct 30 22:13:57.228815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.228768 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.228979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.228914 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.229076 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.229050 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qkp\" (UniqueName: \"kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp\") pod \"payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.229209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.229095 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6tp\" (UniqueName: \"kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp\") pod \"payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.229757 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.229734 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.287040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.287002 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6tp\" (UniqueName: \"kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp\") pod \"payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.330359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.330319 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.330490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.330446 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qkp\" (UniqueName: \"kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp\") pod \"payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.331013 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.330988 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.387005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.386952 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qkp\" (UniqueName: \"kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp\") pod \"payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.474800 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.474772 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:57.495908 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.495883 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:57.855356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.855328 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28"] Oct 30 22:13:57.856287 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:13:57.856265 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac5c48b_8768_4e01_81c2_684c76dab0da.slice/crio-53a444634b1da087d609b3370eebab7337f6568ad8a6cb26adfbb8fe38eb0ea0 WatchSource:0}: Error finding container 53a444634b1da087d609b3370eebab7337f6568ad8a6cb26adfbb8fe38eb0ea0: Status 404 returned error can't find the container with id 53a444634b1da087d609b3370eebab7337f6568ad8a6cb26adfbb8fe38eb0ea0 Oct 30 22:13:57.905196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:57.905174 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7"] Oct 30 22:13:57.907055 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:13:57.907025 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b86dbf_0e5b_4c51_bb04_0a6292f15659.slice/crio-f1d3eb167dbd4d9aa84fc5548826089caad5a210765459abc15b6be00be08117 WatchSource:0}: Error finding container f1d3eb167dbd4d9aa84fc5548826089caad5a210765459abc15b6be00be08117: Status 404 returned error can't find the container with id f1d3eb167dbd4d9aa84fc5548826089caad5a210765459abc15b6be00be08117 Oct 30 22:13:58.418319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.418267 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" event={"ID":"bac5c48b-8768-4e01-81c2-684c76dab0da","Type":"ContainerStarted","Data":"6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c"} Oct 30 22:13:58.418497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.418351 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" event={"ID":"bac5c48b-8768-4e01-81c2-684c76dab0da","Type":"ContainerStarted","Data":"53a444634b1da087d609b3370eebab7337f6568ad8a6cb26adfbb8fe38eb0ea0"} Oct 30 22:13:58.418497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.418386 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:13:58.420027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.419998 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" event={"ID":"81b86dbf-0e5b-4c51-bb04-0a6292f15659","Type":"ContainerStarted","Data":"fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4"} Oct 30 22:13:58.420145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.420040 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" event={"ID":"81b86dbf-0e5b-4c51-bb04-0a6292f15659","Type":"ContainerStarted","Data":"f1d3eb167dbd4d9aa84fc5548826089caad5a210765459abc15b6be00be08117"} Oct 30 22:13:58.420145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.420113 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:13:58.496566 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.496539 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:58.534843 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.534777 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" podStartSLOduration=2.534764388 podStartE2EDuration="2.534764388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:58.532246204 +0000 UTC m=+1848.721783907" watchObservedRunningTime="2026-10-30 22:13:58.534764388 +0000 UTC m=+1848.724302056" Oct 30 22:13:58.669958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:13:58.661819 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" podStartSLOduration=2.6617955479999997 podStartE2EDuration="2.661795548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:58.652206694 +0000 UTC m=+1848.841744396" watchObservedRunningTime="2026-10-30 22:13:58.661795548 +0000 UTC m=+1848.851333225" Oct 30 22:14:00.274644 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:00.274606 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:00.690098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:00.690046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:01.914725 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:01.914687 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs"] Oct 30 22:14:01.919174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:01.919148 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:01.970907 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:01.970876 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5dff71-openshift-default-dockercfg-lfwp9\"" Oct 30 22:14:02.000715 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.000684 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.000922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.000816 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.000922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.000849 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b6b\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.000922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.000891 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.001100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.000950 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.001100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.000970 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.001100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.001048 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.001100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.001097 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.001299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.001119 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.025190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.025153 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs"] Oct 30 22:14:02.028288 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.028256 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:02.116514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116468 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.116720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116632 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.116720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116669 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b6b\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.116720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116713 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.117004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116775 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.117004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116802 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.117004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116927 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.117004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.116999 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.117222 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.117035 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.152755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.130377 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.152755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.130379 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.152755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.130678 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.152755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.134330 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.152755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.135504 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.171443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.171204 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.175603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.175569 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.193218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.193165 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.195326 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.195271 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b6b\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.203100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.203044 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:02.236490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.236429 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:02.447584 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.447478 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:02.682873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.682841 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs"] Oct 30 22:14:02.684135 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:14:02.684108 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9c240d_6eca_4f39_8f16_0ce7a40cf274.slice/crio-417f890df9f077f64b4c0ea55bac98260c6281c221b6361a2061267c4f6bc570 WatchSource:0}: Error finding container 417f890df9f077f64b4c0ea55bac98260c6281c221b6361a2061267c4f6bc570: Status 404 returned error can't find the container with id 417f890df9f077f64b4c0ea55bac98260c6281c221b6361a2061267c4f6bc570 Oct 30 22:14:02.687023 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.686992 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:02.687088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.687060 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:02.687124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:02.687093 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:14:03.468719 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:03.468674 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" event={"ID":"8f9c240d-6eca-4f39-8f16-0ce7a40cf274","Type":"ContainerStarted","Data":"4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343"} Oct 30 22:14:03.469198 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:03.468737 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" event={"ID":"8f9c240d-6eca-4f39-8f16-0ce7a40cf274","Type":"ContainerStarted","Data":"417f890df9f077f64b4c0ea55bac98260c6281c221b6361a2061267c4f6bc570"} Oct 30 22:14:03.636649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:03.636587 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podStartSLOduration=2.636571985 podStartE2EDuration="2.636571985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:03.634526666 +0000 UTC m=+1853.824064379" watchObservedRunningTime="2026-10-30 22:14:03.636571985 +0000 UTC m=+1853.826109660" Oct 30 22:14:04.237867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:04.237829 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:04.243773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:04.243735 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:04.480885 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:04.480848 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:04.481724 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:04.481708 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:14:04.816407 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:04.816369 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:06.413207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:06.413145 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:06.629154 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:06.629118 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:06.873646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:06.873595 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:07.171855 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:07.171764 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.430928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:09.430898 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:14:09.431481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:09.430971 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:14:09.608609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:09.608573 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.848729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:09.848694 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:10.171685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.171594 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95"] Oct 30 22:14:10.199471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.199441 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.264205 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.264163 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95"] Oct 30 22:14:10.280326 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.264999 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-5dff71-dockercfg-wnj4j\"" Oct 30 22:14:10.304745 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.290951 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbm85\" (UniqueName: \"kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85\") pod \"payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.316151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.313571 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.316151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.300492 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-5dff71\"" Oct 30 22:14:10.374094 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:14:10.374061 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:14:10.415143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.415101 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbm85\" (UniqueName: \"kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85\") pod \"payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.415351 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.415275 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.416175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.416147 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.471811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.471733 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbm85\" (UniqueName: \"kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85\") pod \"payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.607692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.607656 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:10.925348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:10.925294 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7"] Oct 30 22:14:11.039540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.034960 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95"] Oct 30 22:14:11.041393 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:14:11.041360 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb0ae00_2005_4a1a_b2d5_738e3718afa1.slice/crio-e81f0f87b0a5d3908a7da86adc36348005efcacb55879987ee64802e4a789911 WatchSource:0}: Error finding container e81f0f87b0a5d3908a7da86adc36348005efcacb55879987ee64802e4a789911: Status 404 returned error can't find the container with id e81f0f87b0a5d3908a7da86adc36348005efcacb55879987ee64802e4a789911 Oct 30 22:14:11.178437 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.178290 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:11.334211 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.334159 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:11.530344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.530230 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" event={"ID":"6bb0ae00-2005-4a1a-b2d5-738e3718afa1","Type":"ContainerStarted","Data":"6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08"} Oct 30 22:14:11.530748 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.530359 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:11.530748 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.530380 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" event={"ID":"6bb0ae00-2005-4a1a-b2d5-738e3718afa1","Type":"ContainerStarted","Data":"e81f0f87b0a5d3908a7da86adc36348005efcacb55879987ee64802e4a789911"} Oct 30 22:14:11.647355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:11.647281 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" podStartSLOduration=1.6472664460000002 podStartE2EDuration="1.647266446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:11.64428549 +0000 UTC m=+1861.833823183" watchObservedRunningTime="2026-10-30 22:14:11.647266446 +0000 UTC m=+1861.836804120" Oct 30 22:14:12.705763 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:12.705724 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.869987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:12.869953 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:13.033327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:13.030451 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:15.926605 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:15.926565 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" podUID="b38fcb34-e5d9-4153-a18e-167062d68e7b" containerName="payload-pre-processing" containerID="cri-o://9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7" gracePeriod=25 Oct 30 22:14:16.209882 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.209851 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:14:16.326858 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.326799 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw\" (UniqueName: \"kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw\") pod \"b38fcb34-e5d9-4153-a18e-167062d68e7b\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " Oct 30 22:14:16.327019 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.326876 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume\") pod \"b38fcb34-e5d9-4153-a18e-167062d68e7b\" (UID: \"b38fcb34-e5d9-4153-a18e-167062d68e7b\") " Oct 30 22:14:16.327296 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.327265 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume" pod "b38fcb34-e5d9-4153-a18e-167062d68e7b" (UID: "b38fcb34-e5d9-4153-a18e-167062d68e7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:16.327474 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.327377 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38fcb34-e5d9-4153-a18e-167062d68e7b-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.329483 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.329458 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw" pod "b38fcb34-e5d9-4153-a18e-167062d68e7b" (UID: "b38fcb34-e5d9-4153-a18e-167062d68e7b"). InnerVolumeSpecName "kube-api-access-k7xkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:16.429139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.429107 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7xkw\" (UniqueName: \"kubernetes.io/projected/b38fcb34-e5d9-4153-a18e-167062d68e7b-kube-api-access-k7xkw\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:16.612941 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.612905 5508 generic.go:356] "Generic (PLEG): container finished" podID="b38fcb34-e5d9-4153-a18e-167062d68e7b" containerID="9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7" exitCode=0 Oct 30 22:14:16.613144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.612950 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" Oct 30 22:14:16.613144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.612988 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" event={"ID":"b38fcb34-e5d9-4153-a18e-167062d68e7b","Type":"ContainerDied","Data":"9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7"} Oct 30 22:14:16.613144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.613078 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7" event={"ID":"b38fcb34-e5d9-4153-a18e-167062d68e7b","Type":"ContainerDied","Data":"e6eb94b9ebe588496d3b246df1748383dee876b3682e3243d1582f7bf25a8cb3"} Oct 30 22:14:16.613144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.613108 5508 scope.go:122] "RemoveContainer" containerID="9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7" Oct 30 22:14:16.623705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.623689 5508 scope.go:122] "RemoveContainer" containerID="9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7" Oct 30 22:14:16.623985 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:14:16.623965 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7\": container with ID starting with 9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7 not found: ID does not exist" containerID="9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7" Oct 30 22:14:16.624050 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.623998 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7"} err="failed to get container status \"9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7\": rpc error: code = NotFound desc = could not find container \"9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7\": container with ID starting with 9ad3344d1632b2a348407d9c61d59db3ad7bf71a6e1dbd4292cf6db4a33753e7 not found: ID does not exist" Oct 30 22:14:16.784519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.784483 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7"] Oct 30 22:14:16.789032 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:16.788999 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2a25a4-57f47dc9b7-f6mx7"] Oct 30 22:14:17.910999 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:17.910962 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.973403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:17.973371 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38fcb34-e5d9-4153-a18e-167062d68e7b" path="/var/lib/kubelet/pods/b38fcb34-e5d9-4153-a18e-167062d68e7b/volumes" Oct 30 22:14:19.932340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:19.931807 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 22:14:20.159443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:20.159383 5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:14:22.580356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:22.580290 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:14:22.914191 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:22.914111 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:27.700655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:27.700609 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.799562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:31.799471 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:32.562733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:32.562690 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:37.802091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:37.802049 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:40.374075 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:14:40.374036 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:14:42.877162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:42.877124 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:45.259940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:45.259898 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:45.583694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:45.583663 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:47.063008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.062974 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5"] Oct 30 22:14:47.063490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.063186 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" podUID="85ebe6c5-3c70-43ba-8a54-060dbf58390a" containerName="maas-api" containerID="cri-o://7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c" gracePeriod=30 Oct 30 22:14:47.209252 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.209219 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:47.388510 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.388482 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:14:47.452140 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.452101 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca\") pod \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " Oct 30 22:14:47.470414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.467466 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls\") pod \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " Oct 30 22:14:47.470414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.467612 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs\") pod \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " Oct 30 22:14:47.470414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.468056 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x\" (UniqueName: \"kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x\") pod \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\" (UID: \"85ebe6c5-3c70-43ba-8a54-060dbf58390a\") " Oct 30 22:14:47.470414 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.463851 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca" pod "85ebe6c5-3c70-43ba-8a54-060dbf58390a" (UID: "85ebe6c5-3c70-43ba-8a54-060dbf58390a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:47.479392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.478827 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs" pod "85ebe6c5-3c70-43ba-8a54-060dbf58390a" (UID: "85ebe6c5-3c70-43ba-8a54-060dbf58390a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:47.529610 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.519208 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x" pod "85ebe6c5-3c70-43ba-8a54-060dbf58390a" (UID: "85ebe6c5-3c70-43ba-8a54-060dbf58390a"). InnerVolumeSpecName "kube-api-access-b9x7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:47.530468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.530434 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls" pod "85ebe6c5-3c70-43ba-8a54-060dbf58390a" (UID: "85ebe6c5-3c70-43ba-8a54-060dbf58390a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:47.572039 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.571739 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9x7x\" (UniqueName: \"kubernetes.io/projected/85ebe6c5-3c70-43ba-8a54-060dbf58390a-kube-api-access-b9x7x\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:47.572039 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.571780 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ebe6c5-3c70-43ba-8a54-060dbf58390a-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:47.572039 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.571796 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:47.572039 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.571809 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ebe6c5-3c70-43ba-8a54-060dbf58390a-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:47.911885 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.911850 5508 generic.go:356] "Generic (PLEG): container finished" podID="85ebe6c5-3c70-43ba-8a54-060dbf58390a" containerID="7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c" exitCode=0 Oct 30 22:14:47.912088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.911905 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" Oct 30 22:14:47.912088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.911935 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" event={"ID":"85ebe6c5-3c70-43ba-8a54-060dbf58390a","Type":"ContainerDied","Data":"7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c"} Oct 30 22:14:47.912088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.912035 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5" event={"ID":"85ebe6c5-3c70-43ba-8a54-060dbf58390a","Type":"ContainerDied","Data":"22e6f40eaafdeace10444680c408651a08f5efb2ba56a165c5974d95b1db19df"} Oct 30 22:14:47.912088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.912066 5508 scope.go:122] "RemoveContainer" containerID="7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c" Oct 30 22:14:47.922402 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.922385 5508 scope.go:122] "RemoveContainer" containerID="7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c" Oct 30 22:14:47.922720 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:14:47.922701 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c\": container with ID starting with 7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c not found: ID does not exist" containerID="7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c" Oct 30 22:14:47.922784 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:47.922733 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c"} err="failed to get container status \"7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c\": rpc error: code = NotFound desc = could not find container \"7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c\": container with ID starting with 7db519904d316f275f334a4ba990e39c43eb971763ad14f46aa2bd6cb11fd00c not found: ID does not exist" Oct 30 22:14:48.135567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:48.135533 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5"] Oct 30 22:14:48.159273 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:48.159241 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-2a25a4-5f9df4b867-dwmp5"] Oct 30 22:14:48.979133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:48.979090 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:49.218242 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:49.218208 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:49.500440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:49.500390 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:49.953634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:49.953603 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ebe6c5-3c70-43ba-8a54-060dbf58390a" path="/var/lib/kubelet/pods/85ebe6c5-3c70-43ba-8a54-060dbf58390a/volumes" Oct 30 22:14:50.157266 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:50.157237 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:51.398041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:51.398006 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:54.221956 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:54.221921 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:56.510586 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:56.510548 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:56.688988 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:56.688935 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:56.861564 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:56.861530 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:59.786870 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:14:59.786833 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:00.230603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.230565 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74"] Oct 30 22:15:00.231292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.231277 5508 cpu_manager.go:414] "containerMap: removing container" podUID="b38fcb34-e5d9-4153-a18e-167062d68e7b" containerName="payload-pre-processing" Oct 30 22:15:00.231359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.231294 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38fcb34-e5d9-4153-a18e-167062d68e7b" containerName="payload-pre-processing" Oct 30 22:15:00.231359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.231336 5508 cpu_manager.go:414] "containerMap: removing container" podUID="85ebe6c5-3c70-43ba-8a54-060dbf58390a" containerName="maas-api" Oct 30 22:15:00.231359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.231344 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ebe6c5-3c70-43ba-8a54-060dbf58390a" containerName="maas-api" Oct 30 22:15:00.231468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.231458 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ebe6c5-3c70-43ba-8a54-060dbf58390a" containerName="maas-api" Oct 30 22:15:00.231502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.231477 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38fcb34-e5d9-4153-a18e-167062d68e7b" containerName="payload-pre-processing" Oct 30 22:15:00.235119 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.235096 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.255971 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.255933 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn"] Oct 30 22:15:00.261652 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.261625 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.276158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.276131 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc"] Oct 30 22:15:00.280696 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.280671 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.311638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.311602 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9"] Oct 30 22:15:00.313559 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.313533 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:15:00.316721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.316702 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.330730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.330696 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5"] Oct 30 22:15:00.335500 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.335472 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.380436 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.380401 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74"] Oct 30 22:15:00.384127 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.384096 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn"] Oct 30 22:15:00.387581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.387518 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc"] Oct 30 22:15:00.390083 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.390056 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwh2v\" (UniqueName: \"kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.390226 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.390124 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.390299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.390254 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.390428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.390399 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5555\" (UniqueName: \"kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555\") pod \"maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.390520 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.390503 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.390602 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.390588 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxd4\" (UniqueName: \"kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.391806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.391783 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz"] Oct 30 22:15:00.402448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.402424 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9"] Oct 30 22:15:00.402698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.402672 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.408221 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.408192 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5"] Oct 30 22:15:00.437095 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.437063 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz"] Oct 30 22:15:00.491873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.491784 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.491873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.491823 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.492082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.491906 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwh2v\" (UniqueName: \"kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.492082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.491985 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24br\" (UniqueName: \"kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.492082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492015 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.492244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492101 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.492244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492155 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5555\" (UniqueName: \"kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555\") pod \"maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.492244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492233 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.492412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492339 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxd4\" (UniqueName: \"kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.492467 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492421 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ms8\" (UniqueName: \"kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.492770 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492747 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.492863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492787 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.492926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.492912 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.553123 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.553091 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5555\" (UniqueName: \"kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555\") pod \"maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.553268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.553194 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxd4\" (UniqueName: \"kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.553268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.553210 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwh2v\" (UniqueName: \"kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.576586 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.576552 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:00.593416 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.593388 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ms8\" (UniqueName: \"kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.593542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.593434 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.593542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.593462 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.593646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.593556 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24br\" (UniqueName: \"kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.593646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.593600 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.593743 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.593696 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9f8m\" (UniqueName: \"kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m\") pod \"maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.594229 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.594204 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.594417 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.594391 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.596317 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.596278 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:00.654705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.654642 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ms8\" (UniqueName: \"kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.656935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.656909 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24br\" (UniqueName: \"kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:00.673479 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.673449 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:00.695041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.694662 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9f8m\" (UniqueName: \"kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m\") pod \"maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.695041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.694941 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.695510 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.695491 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.752239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.752097 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9f8m\" (UniqueName: \"kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m\") pod \"maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:00.851671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.851640 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:00.935756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:00.935723 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:01.006881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.006607 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn"] Oct 30 22:15:01.025264 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.025228 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc"] Oct 30 22:15:01.026112 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.025662 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerStarted","Data":"737bf9071f1f8728bff8e865f246a5d6196e9a01747897b1a7a112f7cb395207"} Oct 30 22:15:01.026217 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.026067 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:01.027236 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:15:01.026739 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeffe2b3_2d29_48b6_99b1_6fd6235c6061.slice/crio-5327071e23f05a554dbebca4fcdd5ee9289aa53b991d8ed13a3f11c05d38cc20 WatchSource:0}: Error finding container 5327071e23f05a554dbebca4fcdd5ee9289aa53b991d8ed13a3f11c05d38cc20: Status 404 returned error can't find the container with id 5327071e23f05a554dbebca4fcdd5ee9289aa53b991d8ed13a3f11c05d38cc20 Oct 30 22:15:01.083817 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.083783 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5"] Oct 30 22:15:01.084122 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:15:01.084092 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb271e6_5de8_433b_bbee_19c54adb327b.slice/crio-e8fa0096bcb1b9b7c7f93467f15ebaa84232ce84a80c9c3bf21485c3186d68cd WatchSource:0}: Error finding container e8fa0096bcb1b9b7c7f93467f15ebaa84232ce84a80c9c3bf21485c3186d68cd: Status 404 returned error can't find the container with id e8fa0096bcb1b9b7c7f93467f15ebaa84232ce84a80c9c3bf21485c3186d68cd Oct 30 22:15:01.240750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.240727 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74"] Oct 30 22:15:01.280616 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:15:01.280551 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444c2438_bb1a_4249_89e9_a14ad8856bdc.slice/crio-ed25b3b1718748774c8f56363b44cb45af250b511b25470d39c25bd9cf224338 WatchSource:0}: Error finding container ed25b3b1718748774c8f56363b44cb45af250b511b25470d39c25bd9cf224338: Status 404 returned error can't find the container with id ed25b3b1718748774c8f56363b44cb45af250b511b25470d39c25bd9cf224338 Oct 30 22:15:01.347390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.347363 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9"] Oct 30 22:15:01.348641 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:15:01.348615 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a33c869_6a90_47f8_87df_3c89dfa0b515.slice/crio-1dfc4343a5117cfc364f2d64e920aa84a93695d0a1feba42eddb36d5c154ca09 WatchSource:0}: Error finding container 1dfc4343a5117cfc364f2d64e920aa84a93695d0a1feba42eddb36d5c154ca09: Status 404 returned error can't find the container with id 1dfc4343a5117cfc364f2d64e920aa84a93695d0a1feba42eddb36d5c154ca09 Oct 30 22:15:01.449353 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:01.449328 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz"] Oct 30 22:15:01.450548 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:15:01.450523 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa7349f_3fdb_42aa_839c_1d7bc8b1a61b.slice/crio-d57084d042c88b4e6a1196bf71536b9608fcd799624b692e6abcdd59ad705846 WatchSource:0}: Error finding container d57084d042c88b4e6a1196bf71536b9608fcd799624b692e6abcdd59ad705846: Status 404 returned error can't find the container with id d57084d042c88b4e6a1196bf71536b9608fcd799624b692e6abcdd59ad705846 Oct 30 22:15:02.035260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.035221 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerStarted","Data":"a08d00d9f5e0b7d5cee929118a568843b011ed7ec5cec25bb860830e679f5b6a"} Oct 30 22:15:02.036930 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.036904 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerStarted","Data":"75b31844ac1d64d824677628b4c5c903cf90f5ce6d7f1b7ce157b9996f8f014e"} Oct 30 22:15:02.037271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.036958 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerStarted","Data":"5327071e23f05a554dbebca4fcdd5ee9289aa53b991d8ed13a3f11c05d38cc20"} Oct 30 22:15:02.038554 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.038527 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerStarted","Data":"83715dcdb79954197fcea0beb0e9cda66d228e31c7fd36d0f4242b97b9e643d0"} Oct 30 22:15:02.038653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.038577 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerStarted","Data":"1dfc4343a5117cfc364f2d64e920aa84a93695d0a1feba42eddb36d5c154ca09"} Oct 30 22:15:02.040030 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.040002 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerStarted","Data":"0a375e0d4daf47f0f86fcce0d32e2cff16a077fef1da988604638a6724f0dc0a"} Oct 30 22:15:02.040151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.040048 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerStarted","Data":"ed25b3b1718748774c8f56363b44cb45af250b511b25470d39c25bd9cf224338"} Oct 30 22:15:02.041658 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.041633 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerStarted","Data":"573c175135dbde7b8be665924a7b750ba91bf6e7a7ea5d894a7ee41957a1b187"} Oct 30 22:15:02.041777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.041676 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerStarted","Data":"e8fa0096bcb1b9b7c7f93467f15ebaa84232ce84a80c9c3bf21485c3186d68cd"} Oct 30 22:15:02.042892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.042873 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" event={"ID":"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b","Type":"ContainerStarted","Data":"d57084d042c88b4e6a1196bf71536b9608fcd799624b692e6abcdd59ad705846"} Oct 30 22:15:02.148613 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.148492 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" podStartSLOduration=2.148477904 podStartE2EDuration="2.148477904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.14672026 +0000 UTC m=+1912.336257945" watchObservedRunningTime="2026-10-30 22:15:02.148477904 +0000 UTC m=+1912.338015578" Oct 30 22:15:02.262145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.262086 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" podStartSLOduration=2.262070219 podStartE2EDuration="2.262070219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.261380737 +0000 UTC m=+1912.450918439" watchObservedRunningTime="2026-10-30 22:15:02.262070219 +0000 UTC m=+1912.451607895" Oct 30 22:15:02.381527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.381469 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" podStartSLOduration=2.381456829 podStartE2EDuration="2.381456829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.377955241 +0000 UTC m=+1912.567492938" watchObservedRunningTime="2026-10-30 22:15:02.381456829 +0000 UTC m=+1912.570994503" Oct 30 22:15:02.496743 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.496687 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" podStartSLOduration=2.496669952 podStartE2EDuration="2.496669952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.496116477 +0000 UTC m=+1912.685654152" watchObservedRunningTime="2026-10-30 22:15:02.496669952 +0000 UTC m=+1912.686207629" Oct 30 22:15:02.611728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:02.611676 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" podStartSLOduration=2.611660957 podStartE2EDuration="2.611660957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.60938517 +0000 UTC m=+1912.798922864" watchObservedRunningTime="2026-10-30 22:15:02.611660957 +0000 UTC m=+1912.801198669" Oct 30 22:15:03.051431 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.051397 5508 generic.go:356] "Generic (PLEG): container finished" podID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerID="0a375e0d4daf47f0f86fcce0d32e2cff16a077fef1da988604638a6724f0dc0a" exitCode=60 Oct 30 22:15:03.051901 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.051487 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerDied","Data":"0a375e0d4daf47f0f86fcce0d32e2cff16a077fef1da988604638a6724f0dc0a"} Oct 30 22:15:03.051901 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.051788 5508 scope.go:122] "RemoveContainer" containerID="0a375e0d4daf47f0f86fcce0d32e2cff16a077fef1da988604638a6724f0dc0a" Oct 30 22:15:03.053289 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.053266 5508 generic.go:356] "Generic (PLEG): container finished" podID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerID="573c175135dbde7b8be665924a7b750ba91bf6e7a7ea5d894a7ee41957a1b187" exitCode=60 Oct 30 22:15:03.053432 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.053340 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerDied","Data":"573c175135dbde7b8be665924a7b750ba91bf6e7a7ea5d894a7ee41957a1b187"} Oct 30 22:15:03.053628 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.053610 5508 scope.go:122] "RemoveContainer" containerID="573c175135dbde7b8be665924a7b750ba91bf6e7a7ea5d894a7ee41957a1b187" Oct 30 22:15:03.055214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.055136 5508 generic.go:356] "Generic (PLEG): container finished" podID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerID="c3e92ebb3c83d70fc126182c8b0c9df132d07fb1ac56a26d80609f06212cd873" exitCode=60 Oct 30 22:15:03.055285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.055252 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" event={"ID":"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b","Type":"ContainerDied","Data":"c3e92ebb3c83d70fc126182c8b0c9df132d07fb1ac56a26d80609f06212cd873"} Oct 30 22:15:03.055546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.055474 5508 scope.go:122] "RemoveContainer" containerID="c3e92ebb3c83d70fc126182c8b0c9df132d07fb1ac56a26d80609f06212cd873" Oct 30 22:15:03.057161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.057141 5508 generic.go:356] "Generic (PLEG): container finished" podID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerID="a08d00d9f5e0b7d5cee929118a568843b011ed7ec5cec25bb860830e679f5b6a" exitCode=60 Oct 30 22:15:03.057235 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.057204 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerDied","Data":"a08d00d9f5e0b7d5cee929118a568843b011ed7ec5cec25bb860830e679f5b6a"} Oct 30 22:15:03.057451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.057436 5508 scope.go:122] "RemoveContainer" containerID="a08d00d9f5e0b7d5cee929118a568843b011ed7ec5cec25bb860830e679f5b6a" Oct 30 22:15:03.058892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.058871 5508 generic.go:356] "Generic (PLEG): container finished" podID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerID="75b31844ac1d64d824677628b4c5c903cf90f5ce6d7f1b7ce157b9996f8f014e" exitCode=60 Oct 30 22:15:03.058987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.058903 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerDied","Data":"75b31844ac1d64d824677628b4c5c903cf90f5ce6d7f1b7ce157b9996f8f014e"} Oct 30 22:15:03.059174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.059154 5508 scope.go:122] "RemoveContainer" containerID="75b31844ac1d64d824677628b4c5c903cf90f5ce6d7f1b7ce157b9996f8f014e" Oct 30 22:15:03.060549 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.060531 5508 generic.go:356] "Generic (PLEG): container finished" podID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerID="83715dcdb79954197fcea0beb0e9cda66d228e31c7fd36d0f4242b97b9e643d0" exitCode=60 Oct 30 22:15:03.060637 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.060594 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerDied","Data":"83715dcdb79954197fcea0beb0e9cda66d228e31c7fd36d0f4242b97b9e643d0"} Oct 30 22:15:03.060832 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:03.060817 5508 scope.go:122] "RemoveContainer" containerID="83715dcdb79954197fcea0beb0e9cda66d228e31c7fd36d0f4242b97b9e643d0" Oct 30 22:15:04.071238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.071204 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerStarted","Data":"838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0"} Oct 30 22:15:04.073243 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.073216 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerStarted","Data":"4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d"} Oct 30 22:15:04.076417 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.076382 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerStarted","Data":"6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f"} Oct 30 22:15:04.079768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.079735 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" event={"ID":"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b","Type":"ContainerStarted","Data":"7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093"} Oct 30 22:15:04.081796 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.081773 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerStarted","Data":"787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c"} Oct 30 22:15:04.083744 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.083715 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerStarted","Data":"bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b"} Oct 30 22:15:04.321696 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:04.321038 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" podStartSLOduration=4.321019765 podStartE2EDuration="4.321019765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.320142004 +0000 UTC m=+1914.509679692" watchObservedRunningTime="2026-10-30 22:15:04.321019765 +0000 UTC m=+1914.510557442" Oct 30 22:15:05.103199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.103159 5508 generic.go:356] "Generic (PLEG): container finished" podID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerID="6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f" exitCode=60 Oct 30 22:15:05.103721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.103249 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerDied","Data":"6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f"} Oct 30 22:15:05.103721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.103414 5508 scope.go:122] "RemoveContainer" containerID="573c175135dbde7b8be665924a7b750ba91bf6e7a7ea5d894a7ee41957a1b187" Oct 30 22:15:05.103721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.103562 5508 scope.go:122] "RemoveContainer" containerID="6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f" Oct 30 22:15:05.103923 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:05.103897 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5_odh-ai-gateway-infra(9bb271e6-5de8-433b-bbee-19c54adb327b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" Oct 30 22:15:05.105844 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.105822 5508 generic.go:356] "Generic (PLEG): container finished" podID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerID="7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093" exitCode=60 Oct 30 22:15:05.105944 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.105891 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" event={"ID":"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b","Type":"ContainerDied","Data":"7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093"} Oct 30 22:15:05.106168 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.106148 5508 scope.go:122] "RemoveContainer" containerID="7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093" Oct 30 22:15:05.106544 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:05.106519 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz_odh-ai-gateway-infra(5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" Oct 30 22:15:05.108028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.108006 5508 generic.go:356] "Generic (PLEG): container finished" podID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerID="787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c" exitCode=60 Oct 30 22:15:05.108139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.108086 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerDied","Data":"787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c"} Oct 30 22:15:05.108350 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.108334 5508 scope.go:122] "RemoveContainer" containerID="787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c" Oct 30 22:15:05.108667 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:05.108644 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn_odh-ai-gateway-infra(8d1257b3-9261-404f-955e-65b2aff82ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" Oct 30 22:15:05.110295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.110273 5508 generic.go:356] "Generic (PLEG): container finished" podID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerID="bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b" exitCode=60 Oct 30 22:15:05.110408 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.110361 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerDied","Data":"bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b"} Oct 30 22:15:05.110638 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.110624 5508 scope.go:122] "RemoveContainer" containerID="bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b" Oct 30 22:15:05.110909 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:05.110889 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc_odh-ai-gateway-infra(beffe2b3-2d29-48b6-99b1-6fd6235c6061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" Oct 30 22:15:05.112440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.112418 5508 generic.go:356] "Generic (PLEG): container finished" podID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerID="838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0" exitCode=60 Oct 30 22:15:05.112578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.112548 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerDied","Data":"838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0"} Oct 30 22:15:05.112721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.112704 5508 scope.go:122] "RemoveContainer" containerID="838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0" Oct 30 22:15:05.113017 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:05.112992 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9_odh-ai-gateway-infra(6a33c869-6a90-47f8-87df-3c89dfa0b515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" Oct 30 22:15:05.114464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.114442 5508 generic.go:356] "Generic (PLEG): container finished" podID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerID="4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d" exitCode=60 Oct 30 22:15:05.114697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.114479 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerDied","Data":"4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d"} Oct 30 22:15:05.115028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.114700 5508 scope.go:122] "RemoveContainer" containerID="4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d" Oct 30 22:15:05.115028 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:05.114914 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74_odh-ai-gateway-infra(444c2438-bb1a-4249-89e9-a14ad8856bdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" Oct 30 22:15:05.118848 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.118830 5508 scope.go:122] "RemoveContainer" containerID="c3e92ebb3c83d70fc126182c8b0c9df132d07fb1ac56a26d80609f06212cd873" Oct 30 22:15:05.132626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.132603 5508 scope.go:122] "RemoveContainer" containerID="a08d00d9f5e0b7d5cee929118a568843b011ed7ec5cec25bb860830e679f5b6a" Oct 30 22:15:05.146164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.146141 5508 scope.go:122] "RemoveContainer" containerID="75b31844ac1d64d824677628b4c5c903cf90f5ce6d7f1b7ce157b9996f8f014e" Oct 30 22:15:05.162796 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.162770 5508 scope.go:122] "RemoveContainer" containerID="83715dcdb79954197fcea0beb0e9cda66d228e31c7fd36d0f4242b97b9e643d0" Oct 30 22:15:05.176132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.176111 5508 scope.go:122] "RemoveContainer" containerID="0a375e0d4daf47f0f86fcce0d32e2cff16a077fef1da988604638a6724f0dc0a" Oct 30 22:15:05.283867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:05.283811 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.123862 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:06.123834 5508 scope.go:122] "RemoveContainer" containerID="7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093" Oct 30 22:15:06.124334 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:06.124197 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz_odh-ai-gateway-infra(5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" Oct 30 22:15:06.125620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:06.125602 5508 scope.go:122] "RemoveContainer" containerID="787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c" Oct 30 22:15:06.125941 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:06.125921 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn_odh-ai-gateway-infra(8d1257b3-9261-404f-955e-65b2aff82ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" Oct 30 22:15:06.127155 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:06.127135 5508 scope.go:122] "RemoveContainer" containerID="bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b" Oct 30 22:15:06.127460 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:06.127440 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc_odh-ai-gateway-infra(beffe2b3-2d29-48b6-99b1-6fd6235c6061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" Oct 30 22:15:06.128653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:06.128635 5508 scope.go:122] "RemoveContainer" containerID="838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0" Oct 30 22:15:06.128920 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:06.128902 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9_odh-ai-gateway-infra(6a33c869-6a90-47f8-87df-3c89dfa0b515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" Oct 30 22:15:06.130122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:06.130103 5508 scope.go:122] "RemoveContainer" containerID="4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d" Oct 30 22:15:06.130358 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:06.130337 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74_odh-ai-gateway-infra(444c2438-bb1a-4249-89e9-a14ad8856bdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" Oct 30 22:15:06.131668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:06.131650 5508 scope.go:122] "RemoveContainer" containerID="6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f" Oct 30 22:15:06.131857 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:06.131842 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5_odh-ai-gateway-infra(9bb271e6-5de8-433b-bbee-19c54adb327b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" Oct 30 22:15:09.231119 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:09.231084 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:09.403276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:09.403232 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:09.721728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:09.721688 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:09.881514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:09.881467 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:10.374989 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:10.374954 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:15:10.604879 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:10.604834 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:10.767004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:10.766929 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:14.945687 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:14.945648 5508 scope.go:122] "RemoveContainer" containerID="6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f" Oct 30 22:15:16.204843 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.204753 5508 generic.go:356] "Generic (PLEG): container finished" podID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerID="ef7c6c2cace9ae6a7dd79561642bbe69d499ad71031fa3354803b9e5ea4bb523" exitCode=60 Oct 30 22:15:16.204843 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.204822 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerDied","Data":"ef7c6c2cace9ae6a7dd79561642bbe69d499ad71031fa3354803b9e5ea4bb523"} Oct 30 22:15:16.205404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.204898 5508 scope.go:122] "RemoveContainer" containerID="6f94944312ffae1a658f3fd37bfab43be267be6f189d65201a3b2d0c06f0204f" Oct 30 22:15:16.205460 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.205424 5508 scope.go:122] "RemoveContainer" containerID="ef7c6c2cace9ae6a7dd79561642bbe69d499ad71031fa3354803b9e5ea4bb523" Oct 30 22:15:16.205784 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:16.205762 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5_odh-ai-gateway-infra(9bb271e6-5de8-433b-bbee-19c54adb327b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" Oct 30 22:15:16.300505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.300468 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:16.945698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.945670 5508 scope.go:122] "RemoveContainer" containerID="4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d" Oct 30 22:15:16.945900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.945834 5508 scope.go:122] "RemoveContainer" containerID="787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c" Oct 30 22:15:16.945982 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.945948 5508 scope.go:122] "RemoveContainer" containerID="bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b" Oct 30 22:15:16.946036 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:16.946007 5508 scope.go:122] "RemoveContainer" containerID="838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0" Oct 30 22:15:17.330475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.330442 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5"] Oct 30 22:15:17.514116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.514087 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:17.651000 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.650971 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8\" (UniqueName: \"kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8\") pod \"9bb271e6-5de8-433b-bbee-19c54adb327b\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " Oct 30 22:15:17.651196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.651031 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca\") pod \"9bb271e6-5de8-433b-bbee-19c54adb327b\" (UID: \"9bb271e6-5de8-433b-bbee-19c54adb327b\") " Oct 30 22:15:17.651431 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.651410 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca" pod "9bb271e6-5de8-433b-bbee-19c54adb327b" (UID: "9bb271e6-5de8-433b-bbee-19c54adb327b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:17.651643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.651624 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb271e6-5de8-433b-bbee-19c54adb327b-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.653537 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.653513 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8" pod "9bb271e6-5de8-433b-bbee-19c54adb327b" (UID: "9bb271e6-5de8-433b-bbee-19c54adb327b"). InnerVolumeSpecName "kube-api-access-42ms8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:17.753521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.753478 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ms8\" (UniqueName: \"kubernetes.io/projected/9bb271e6-5de8-433b-bbee-19c54adb327b-kube-api-access-42ms8\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.945683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:17.945593 5508 scope.go:122] "RemoveContainer" containerID="7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093" Oct 30 22:15:18.223688 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.223659 5508 generic.go:356] "Generic (PLEG): container finished" podID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerID="25e240ce04fa420bb4e3f543434dba2f9162b4c723e7cb48fafcf1c48580d9bc" exitCode=60 Oct 30 22:15:18.223827 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.223735 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerDied","Data":"25e240ce04fa420bb4e3f543434dba2f9162b4c723e7cb48fafcf1c48580d9bc"} Oct 30 22:15:18.223827 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.223797 5508 scope.go:122] "RemoveContainer" containerID="bbea1150b2e10d5477403cb65519343bce2787c3c014389ef6fa0e31cd38b36b" Oct 30 22:15:18.224242 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.224221 5508 scope.go:122] "RemoveContainer" containerID="25e240ce04fa420bb4e3f543434dba2f9162b4c723e7cb48fafcf1c48580d9bc" Oct 30 22:15:18.224638 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:18.224615 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc_odh-ai-gateway-infra(beffe2b3-2d29-48b6-99b1-6fd6235c6061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" Oct 30 22:15:18.225987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.225967 5508 generic.go:356] "Generic (PLEG): container finished" podID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerID="9c49599c92484ff429ddfee8ddc4d9ed1dc45afde17b2c056b3df154ddabf5df" exitCode=60 Oct 30 22:15:18.226072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.226035 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerDied","Data":"9c49599c92484ff429ddfee8ddc4d9ed1dc45afde17b2c056b3df154ddabf5df"} Oct 30 22:15:18.226480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.226452 5508 scope.go:122] "RemoveContainer" containerID="9c49599c92484ff429ddfee8ddc4d9ed1dc45afde17b2c056b3df154ddabf5df" Oct 30 22:15:18.226836 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:18.226815 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9_odh-ai-gateway-infra(6a33c869-6a90-47f8-87df-3c89dfa0b515)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" Oct 30 22:15:18.228118 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.228096 5508 generic.go:356] "Generic (PLEG): container finished" podID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerID="7e60392a4312dad5a73b39543e90795e0cd6eefb65b6996c5b09d1ccb78319b0" exitCode=60 Oct 30 22:15:18.228198 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.228161 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerDied","Data":"7e60392a4312dad5a73b39543e90795e0cd6eefb65b6996c5b09d1ccb78319b0"} Oct 30 22:15:18.228534 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.228519 5508 scope.go:122] "RemoveContainer" containerID="7e60392a4312dad5a73b39543e90795e0cd6eefb65b6996c5b09d1ccb78319b0" Oct 30 22:15:18.228829 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:18.228806 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74_odh-ai-gateway-infra(444c2438-bb1a-4249-89e9-a14ad8856bdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" Oct 30 22:15:18.229573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.229557 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" Oct 30 22:15:18.229671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.229604 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5" event={"ID":"9bb271e6-5de8-433b-bbee-19c54adb327b","Type":"ContainerDied","Data":"e8fa0096bcb1b9b7c7f93467f15ebaa84232ce84a80c9c3bf21485c3186d68cd"} Oct 30 22:15:18.231502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.231482 5508 generic.go:356] "Generic (PLEG): container finished" podID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerID="7522253800ce7d70930eb9e2532f679ac4bbdad9f72564f218190e582631f178" exitCode=60 Oct 30 22:15:18.231573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.231506 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerDied","Data":"7522253800ce7d70930eb9e2532f679ac4bbdad9f72564f218190e582631f178"} Oct 30 22:15:18.231776 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.231764 5508 scope.go:122] "RemoveContainer" containerID="7522253800ce7d70930eb9e2532f679ac4bbdad9f72564f218190e582631f178" Oct 30 22:15:18.231983 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:18.231968 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn_odh-ai-gateway-infra(8d1257b3-9261-404f-955e-65b2aff82ee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" Oct 30 22:15:18.357344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.357158 5508 scope.go:122] "RemoveContainer" containerID="838742958a60061599b9f691c34417570975fd2b8951ece15d6f48de5419f5f0" Oct 30 22:15:18.374564 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.374544 5508 scope.go:122] "RemoveContainer" containerID="4fd0c124c2b9a970aaa15882955497d62e3ce18c941683590d5ad2e1b173ab2d" Oct 30 22:15:18.387814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.387751 5508 scope.go:122] "RemoveContainer" containerID="ef7c6c2cace9ae6a7dd79561642bbe69d499ad71031fa3354803b9e5ea4bb523" Oct 30 22:15:18.396658 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.396638 5508 scope.go:122] "RemoveContainer" containerID="787f15f3dfb515deaa024da59eadf059077827725414256435e51cf613e8393c" Oct 30 22:15:18.846987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.846953 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5"] Oct 30 22:15:18.856458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:18.856429 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889975-h8kq5"] Oct 30 22:15:19.252009 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.251930 5508 generic.go:356] "Generic (PLEG): container finished" podID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerID="55b4ecdd8b8793efd2533de2c7d6167882026cb36948895a738ceb947a8ec06d" exitCode=60 Oct 30 22:15:19.252009 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.251981 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" event={"ID":"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b","Type":"ContainerDied","Data":"55b4ecdd8b8793efd2533de2c7d6167882026cb36948895a738ceb947a8ec06d"} Oct 30 22:15:19.252201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.252042 5508 scope.go:122] "RemoveContainer" containerID="7987b3fe2d01faa3e036f3a432c25ab439d76982fe7aaa6b91c4496e8b14c093" Oct 30 22:15:19.252538 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.252510 5508 scope.go:122] "RemoveContainer" containerID="55b4ecdd8b8793efd2533de2c7d6167882026cb36948895a738ceb947a8ec06d" Oct 30 22:15:19.252922 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:19.252884 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz_odh-ai-gateway-infra(5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" Oct 30 22:15:19.350652 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.350613 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc"] Oct 30 22:15:19.461507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.461476 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74"] Oct 30 22:15:19.560128 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.560100 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:19.578592 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.578557 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9"] Oct 30 22:15:19.686392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.684021 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555\" (UniqueName: \"kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555\") pod \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " Oct 30 22:15:19.686392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.684198 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca\") pod \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\" (UID: \"beffe2b3-2d29-48b6-99b1-6fd6235c6061\") " Oct 30 22:15:19.686392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.685428 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca" pod "beffe2b3-2d29-48b6-99b1-6fd6235c6061" (UID: "beffe2b3-2d29-48b6-99b1-6fd6235c6061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.686392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.686205 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beffe2b3-2d29-48b6-99b1-6fd6235c6061-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.689028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.689004 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555" pod "beffe2b3-2d29-48b6-99b1-6fd6235c6061" (UID: "beffe2b3-2d29-48b6-99b1-6fd6235c6061"). InnerVolumeSpecName "kube-api-access-k5555". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.706460 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.706166 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:19.736774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.736744 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:19.746443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.746419 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn"] Oct 30 22:15:19.787366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.787282 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4\" (UniqueName: \"kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4\") pod \"444c2438-bb1a-4249-89e9-a14ad8856bdc\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " Oct 30 22:15:19.787493 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.787387 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca\") pod \"444c2438-bb1a-4249-89e9-a14ad8856bdc\" (UID: \"444c2438-bb1a-4249-89e9-a14ad8856bdc\") " Oct 30 22:15:19.787886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.787872 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5555\" (UniqueName: \"kubernetes.io/projected/beffe2b3-2d29-48b6-99b1-6fd6235c6061-kube-api-access-k5555\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.788160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.788144 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca" pod "444c2438-bb1a-4249-89e9-a14ad8856bdc" (UID: "444c2438-bb1a-4249-89e9-a14ad8856bdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.790385 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.790358 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4" pod "444c2438-bb1a-4249-89e9-a14ad8856bdc" (UID: "444c2438-bb1a-4249-89e9-a14ad8856bdc"). InnerVolumeSpecName "kube-api-access-mbxd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.888494 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.888468 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br\" (UniqueName: \"kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br\") pod \"6a33c869-6a90-47f8-87df-3c89dfa0b515\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " Oct 30 22:15:19.888652 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.888551 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca\") pod \"6a33c869-6a90-47f8-87df-3c89dfa0b515\" (UID: \"6a33c869-6a90-47f8-87df-3c89dfa0b515\") " Oct 30 22:15:19.889403 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.889363 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca" pod "6a33c869-6a90-47f8-87df-3c89dfa0b515" (UID: "6a33c869-6a90-47f8-87df-3c89dfa0b515"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.889536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.889442 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxd4\" (UniqueName: \"kubernetes.io/projected/444c2438-bb1a-4249-89e9-a14ad8856bdc-kube-api-access-mbxd4\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.889536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.889466 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/444c2438-bb1a-4249-89e9-a14ad8856bdc-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.891602 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.891576 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br" pod "6a33c869-6a90-47f8-87df-3c89dfa0b515" (UID: "6a33c869-6a90-47f8-87df-3c89dfa0b515"). InnerVolumeSpecName "kube-api-access-v24br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.892499 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.892473 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:19.969383 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.968275 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" path="/var/lib/kubelet/pods/9bb271e6-5de8-433b-bbee-19c54adb327b/volumes" Oct 30 22:15:19.990442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.990418 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca\") pod \"8d1257b3-9261-404f-955e-65b2aff82ee2\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " Oct 30 22:15:19.990567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.990514 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v\" (UniqueName: \"kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v\") pod \"8d1257b3-9261-404f-955e-65b2aff82ee2\" (UID: \"8d1257b3-9261-404f-955e-65b2aff82ee2\") " Oct 30 22:15:19.990791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.990763 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca" pod "8d1257b3-9261-404f-955e-65b2aff82ee2" (UID: "8d1257b3-9261-404f-955e-65b2aff82ee2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.991040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.991022 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a33c869-6a90-47f8-87df-3c89dfa0b515-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.991109 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.991045 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1257b3-9261-404f-955e-65b2aff82ee2-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.991109 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.991059 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v24br\" (UniqueName: \"kubernetes.io/projected/6a33c869-6a90-47f8-87df-3c89dfa0b515-kube-api-access-v24br\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.992837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:19.992815 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v" pod "8d1257b3-9261-404f-955e-65b2aff82ee2" (UID: "8d1257b3-9261-404f-955e-65b2aff82ee2"). InnerVolumeSpecName "kube-api-access-gwh2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.092865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.092830 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwh2v\" (UniqueName: \"kubernetes.io/projected/8d1257b3-9261-404f-955e-65b2aff82ee2-kube-api-access-gwh2v\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.261450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.261414 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" event={"ID":"8d1257b3-9261-404f-955e-65b2aff82ee2","Type":"ContainerDied","Data":"737bf9071f1f8728bff8e865f246a5d6196e9a01747897b1a7a112f7cb395207"} Oct 30 22:15:20.261659 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.261431 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn" Oct 30 22:15:20.261659 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.261485 5508 scope.go:122] "RemoveContainer" containerID="7522253800ce7d70930eb9e2532f679ac4bbdad9f72564f218190e582631f178" Oct 30 22:15:20.263064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.263043 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" Oct 30 22:15:20.263374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.263046 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc" event={"ID":"beffe2b3-2d29-48b6-99b1-6fd6235c6061","Type":"ContainerDied","Data":"5327071e23f05a554dbebca4fcdd5ee9289aa53b991d8ed13a3f11c05d38cc20"} Oct 30 22:15:20.264822 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.264795 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" Oct 30 22:15:20.264971 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.264792 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9" event={"ID":"6a33c869-6a90-47f8-87df-3c89dfa0b515","Type":"ContainerDied","Data":"1dfc4343a5117cfc364f2d64e920aa84a93695d0a1feba42eddb36d5c154ca09"} Oct 30 22:15:20.266622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.266604 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" Oct 30 22:15:20.266728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.266625 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74" event={"ID":"444c2438-bb1a-4249-89e9-a14ad8856bdc","Type":"ContainerDied","Data":"ed25b3b1718748774c8f56363b44cb45af250b511b25470d39c25bd9cf224338"} Oct 30 22:15:20.276001 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.275986 5508 scope.go:122] "RemoveContainer" containerID="25e240ce04fa420bb4e3f543434dba2f9162b4c723e7cb48fafcf1c48580d9bc" Oct 30 22:15:20.286575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.286556 5508 scope.go:122] "RemoveContainer" containerID="9c49599c92484ff429ddfee8ddc4d9ed1dc45afde17b2c056b3df154ddabf5df" Oct 30 22:15:20.299985 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.299957 5508 scope.go:122] "RemoveContainer" containerID="7e60392a4312dad5a73b39543e90795e0cd6eefb65b6996c5b09d1ccb78319b0" Oct 30 22:15:20.391226 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.391146 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz"] Oct 30 22:15:20.429714 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.429678 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn"] Oct 30 22:15:20.438019 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.436433 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889975-rcfhn"] Oct 30 22:15:20.540094 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.540063 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc"] Oct 30 22:15:20.545017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.544990 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0c99b1-29889975-qbglc"] Oct 30 22:15:20.581038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.581006 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:20.627401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.627367 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m\" (UniqueName: \"kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m\") pod \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " Oct 30 22:15:20.627563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.627429 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca\") pod \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\" (UID: \"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b\") " Oct 30 22:15:20.627815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.627793 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca" pod "5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" (UID: "5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.628120 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.628102 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.630063 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.630038 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m" pod "5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" (UID: "5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b"). InnerVolumeSpecName "kube-api-access-f9f8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.652399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.652371 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74"] Oct 30 22:15:20.654344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.654298 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889975-smf74"] Oct 30 22:15:20.729599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.729560 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9f8m\" (UniqueName: \"kubernetes.io/projected/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b-kube-api-access-f9f8m\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.768198 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.768164 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9"] Oct 30 22:15:20.775930 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:20.775898 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ebddcc-29889975-dknk9"] Oct 30 22:15:21.280542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.280507 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" event={"ID":"5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b","Type":"ContainerDied","Data":"d57084d042c88b4e6a1196bf71536b9608fcd799624b692e6abcdd59ad705846"} Oct 30 22:15:21.280717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.280548 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz" Oct 30 22:15:21.280717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.280591 5508 scope.go:122] "RemoveContainer" containerID="55b4ecdd8b8793efd2533de2c7d6167882026cb36948895a738ceb947a8ec06d" Oct 30 22:15:21.447587 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.447557 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz"] Oct 30 22:15:21.451795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.451765 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-242ddaf0-29889975-sx6cz"] Oct 30 22:15:21.780547 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.780512 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:21.955146 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.955113 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" path="/var/lib/kubelet/pods/444c2438-bb1a-4249-89e9-a14ad8856bdc/volumes" Oct 30 22:15:21.955749 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.955732 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" path="/var/lib/kubelet/pods/5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b/volumes" Oct 30 22:15:21.956285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.956270 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" path="/var/lib/kubelet/pods/6a33c869-6a90-47f8-87df-3c89dfa0b515/volumes" Oct 30 22:15:21.957349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.957331 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" path="/var/lib/kubelet/pods/8d1257b3-9261-404f-955e-65b2aff82ee2/volumes" Oct 30 22:15:21.957870 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:21.957854 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" path="/var/lib/kubelet/pods/beffe2b3-2d29-48b6-99b1-6fd6235c6061/volumes" Oct 30 22:15:24.654490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:24.654457 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:24.824755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:24.824716 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:27.171273 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:27.171240 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:28.645471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:28.645422 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:30.495527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:30.495480 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:32.146766 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:32.146727 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:32.315552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:32.315505 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:32.482513 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:32.482415 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:33.592850 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:33.592816 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp"] Oct 30 22:15:33.771539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:33.771497 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:37.642760 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:37.642727 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:40.374463 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:40.374422 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:15:40.609972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:40.609935 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf"] Oct 30 22:15:41.123760 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:41.123720 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:41.311835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:41.311796 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:43.442739 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:43.442703 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:45.610287 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:45.610239 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerName="payload-processing" containerID="cri-o://7beab5db64d3e506929f25cffe6a97180312fcbdd37dc0d2d6ff943bd7788395" gracePeriod=25 Oct 30 22:15:48.067413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.066912 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn"] Oct 30 22:15:48.067413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.067166 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" podUID="cf7694c5-5785-42c3-90a8-858c708f7904" containerName="maas-api" containerID="cri-o://abc5aa4c706123a25646965d153801e164d0807c5e8c4bb4914d317d0a9a7403" gracePeriod=30 Oct 30 22:15:48.217992 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.217916 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:48.572355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.572036 5508 generic.go:356] "Generic (PLEG): container finished" podID="cf7694c5-5785-42c3-90a8-858c708f7904" containerID="abc5aa4c706123a25646965d153801e164d0807c5e8c4bb4914d317d0a9a7403" exitCode=0 Oct 30 22:15:48.572355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.572254 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" event={"ID":"cf7694c5-5785-42c3-90a8-858c708f7904","Type":"ContainerDied","Data":"abc5aa4c706123a25646965d153801e164d0807c5e8c4bb4914d317d0a9a7403"} Oct 30 22:15:48.624390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.623720 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" containerID="cri-o://1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a" gracePeriod=45 Oct 30 22:15:48.734187 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.734157 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:15:48.848834 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.848796 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\") pod \"cf7694c5-5785-42c3-90a8-858c708f7904\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " Oct 30 22:15:48.849007 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.848842 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs\") pod \"cf7694c5-5785-42c3-90a8-858c708f7904\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " Oct 30 22:15:48.849007 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.848978 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv\" (UniqueName: \"kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv\") pod \"cf7694c5-5785-42c3-90a8-858c708f7904\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " Oct 30 22:15:48.849135 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.849085 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca\") pod \"cf7694c5-5785-42c3-90a8-858c708f7904\" (UID: \"cf7694c5-5785-42c3-90a8-858c708f7904\") " Oct 30 22:15:48.850097 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.850053 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca" pod "cf7694c5-5785-42c3-90a8-858c708f7904" (UID: "cf7694c5-5785-42c3-90a8-858c708f7904"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:48.859512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.856401 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv" pod "cf7694c5-5785-42c3-90a8-858c708f7904" (UID: "cf7694c5-5785-42c3-90a8-858c708f7904"). InnerVolumeSpecName "kube-api-access-96vsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:48.859512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.857452 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs" pod "cf7694c5-5785-42c3-90a8-858c708f7904" (UID: "cf7694c5-5785-42c3-90a8-858c708f7904"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:48.874889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.874820 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls" pod "cf7694c5-5785-42c3-90a8-858c708f7904" (UID: "cf7694c5-5785-42c3-90a8-858c708f7904"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:48.902062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.902037 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:15:48.950708 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.950674 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache\") pod \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " Oct 30 22:15:48.950903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.950792 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h\" (UniqueName: \"kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h\") pod \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " Oct 30 22:15:48.950903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.950828 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir\") pod \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " Oct 30 22:15:48.950903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.950869 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home\") pod \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " Oct 30 22:15:48.951072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.950944 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm\") pod \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " Oct 30 22:15:48.951072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.950974 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache" pod "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" (UID: "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:48.951177 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951082 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs\") pod \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\" (UID: \"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4\") " Oct 30 22:15:48.951177 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951102 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir" pod "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" (UID: "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:48.951282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951192 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home" pod "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" (UID: "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:48.951959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951934 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.951959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951964 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.952183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951982 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf7694c5-5785-42c3-90a8-858c708f7904-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.952183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.951997 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.952183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.952012 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.952183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.952026 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vsv\" (UniqueName: \"kubernetes.io/projected/cf7694c5-5785-42c3-90a8-858c708f7904-kube-api-access-96vsv\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.952183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.952041 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf7694c5-5785-42c3-90a8-858c708f7904-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:48.953478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.953450 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm" pod "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" (UID: "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:48.953573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.953505 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h" pod "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" (UID: "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4"). InnerVolumeSpecName "kube-api-access-hl24h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:48.953634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:48.953576 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs" pod "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" (UID: "fc6e41a3-3330-4d4d-afb0-7c175c53f4d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:49.066453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.056123 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl24h\" (UniqueName: \"kubernetes.io/projected/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-kube-api-access-hl24h\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.066453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.056164 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.066453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.056180 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.582959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.582927 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" Oct 30 22:15:49.585088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.585035 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn" event={"ID":"cf7694c5-5785-42c3-90a8-858c708f7904","Type":"ContainerDied","Data":"ba5b2bd8b6c70a5056d514daacf5c6320ab0dea78f0d3b12fee393b8f7c9bbfc"} Oct 30 22:15:49.585224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.585104 5508 scope.go:122] "RemoveContainer" containerID="abc5aa4c706123a25646965d153801e164d0807c5e8c4bb4914d317d0a9a7403" Oct 30 22:15:49.586596 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.586566 5508 generic.go:356] "Generic (PLEG): container finished" podID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerID="1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a" exitCode=0 Oct 30 22:15:49.586821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.586801 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" event={"ID":"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4","Type":"ContainerDied","Data":"1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a"} Oct 30 22:15:49.586925 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.586837 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" event={"ID":"fc6e41a3-3330-4d4d-afb0-7c175c53f4d4","Type":"ContainerDied","Data":"a7cb12ba1d85fb80dc09f04e4500b716a94663f57d176234d4ff27641250e9f6"} Oct 30 22:15:49.586925 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.586879 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" Oct 30 22:15:49.606775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.606750 5508 scope.go:122] "RemoveContainer" containerID="1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a" Oct 30 22:15:49.617714 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.617062 5508 scope.go:122] "RemoveContainer" containerID="1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a" Oct 30 22:15:49.617714 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:15:49.617463 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a\": container with ID starting with 1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a not found: ID does not exist" containerID="1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a" Oct 30 22:15:49.617714 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.617503 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a"} err="failed to get container status \"1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a\": rpc error: code = NotFound desc = could not find container \"1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a\": container with ID starting with 1760fd666e08dcb302b01fab183d665644c5dfe227901a465a19215cca5ceb3a not found: ID does not exist" Oct 30 22:15:49.751954 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.751924 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp"] Oct 30 22:15:49.800494 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.800462 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp"] Oct 30 22:15:49.862106 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.862019 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn"] Oct 30 22:15:49.865777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.865731 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:49.868485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.868455 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f1d78db-644bbc9c84-kz8nn"] Oct 30 22:15:49.882480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.882445 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/premium-w1-d516c4-kserve-85df847db4-f7pkp" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 22:15:49.954093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.954052 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7694c5-5785-42c3-90a8-858c708f7904" path="/var/lib/kubelet/pods/cf7694c5-5785-42c3-90a8-858c708f7904/volumes" Oct 30 22:15:49.957487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:49.955959 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" path="/var/lib/kubelet/pods/fc6e41a3-3330-4d4d-afb0-7c175c53f4d4/volumes" Oct 30 22:15:50.168257 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:50.168169 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:50.329176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:50.329138 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:50.836262 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:50.836225 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:51.830041 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:51.829999 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:53.618083 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:53.618049 5508 generic.go:356] "Generic (PLEG): container finished" podID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerID="dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976" exitCode=0 Oct 30 22:15:53.618551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:53.618115 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" event={"ID":"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9","Type":"ContainerDied","Data":"dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976"} Oct 30 22:15:53.618551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:53.618507 5508 scope.go:122] "RemoveContainer" containerID="dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976" Oct 30 22:15:53.620050 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:53.620024 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:53.620113 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:53.620094 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:53.620150 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:53.620130 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:15:54.627429 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:54.627394 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" event={"ID":"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9","Type":"ContainerStarted","Data":"10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079"} Oct 30 22:15:54.655158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:54.655127 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:15:54.655379 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:54.655356 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:54.655436 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:54.655404 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:15:55.600206 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:55.600174 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:55.655990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:55.655801 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:55.655990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:55.655860 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:15:56.656140 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:56.656106 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:56.656563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:56.656169 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:15:57.501361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:57.498173 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:57.662124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:57.660953 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:57.662124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:57.661027 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:15:57.667579 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:57.667539 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:57.842404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:57.842372 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:58.273533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:58.273458 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:58.676331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:58.673155 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:58.676331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:58.673227 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:15:59.595958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:59.595860 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:59.655862 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:59.655829 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:59.656046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:59.655885 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:15:59.757511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:15:59.757468 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:00.660448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:00.660418 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:16:00.660895 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:00.660879 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:16:00.661775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:00.661756 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:16:00.894524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:00.894489 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44"] Oct 30 22:16:00.894896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:00.894698 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" containerID="cri-o://cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d" gracePeriod=30 Oct 30 22:16:01.238975 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:01.238932 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:01.830672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:01.830635 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:16:02.213652 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:02.213561 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:02.213652 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:02.213630 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:03.208063 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:03.208017 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl"] Oct 30 22:16:03.257032 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:03.256994 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz"] Oct 30 22:16:03.400756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:03.400721 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:05.744531 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.744496 5508 generic.go:356] "Generic (PLEG): container finished" podID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerID="7beab5db64d3e506929f25cffe6a97180312fcbdd37dc0d2d6ff943bd7788395" exitCode=0 Oct 30 22:16:05.745062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.744569 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" event={"ID":"f1f6e1fe-0ef9-4e24-a265-ea35721222da","Type":"ContainerDied","Data":"7beab5db64d3e506929f25cffe6a97180312fcbdd37dc0d2d6ff943bd7788395"} Oct 30 22:16:05.891488 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.891430 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 22:16:05.995186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.995156 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp\" (UniqueName: \"kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp\") pod \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " Oct 30 22:16:05.995382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.995325 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume\") pod \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\" (UID: \"f1f6e1fe-0ef9-4e24-a265-ea35721222da\") " Oct 30 22:16:05.995689 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.995663 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume" pod "f1f6e1fe-0ef9-4e24-a265-ea35721222da" (UID: "f1f6e1fe-0ef9-4e24-a265-ea35721222da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:05.995934 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.995915 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f6e1fe-0ef9-4e24-a265-ea35721222da-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:05.997538 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:05.997516 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp" pod "f1f6e1fe-0ef9-4e24-a265-ea35721222da" (UID: "f1f6e1fe-0ef9-4e24-a265-ea35721222da"). InnerVolumeSpecName "kube-api-access-qq2sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:06.097066 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.097021 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2sp\" (UniqueName: \"kubernetes.io/projected/f1f6e1fe-0ef9-4e24-a265-ea35721222da-kube-api-access-qq2sp\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.168585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.168558 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:16:06.299141 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299103 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.299347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299236 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.299432 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299373 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.299432 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299417 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.299557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299494 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.299847 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299816 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:06.300125 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.299987 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:06.300275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.300114 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:06.300656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.300634 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.300759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.300703 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.300759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.300748 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.300884 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.300812 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo\") pod \"30a1d1b2-c64b-4775-ac0c-115289b715d8\" (UID: \"30a1d1b2-c64b-4775-ac0c-115289b715d8\") " Oct 30 22:16:06.301572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.301292 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:06.301668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.301648 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.301733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.301674 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.301733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.301690 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.301733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.301707 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a1d1b2-c64b-4775-ac0c-115289b715d8-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.301971 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.301937 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:06.302755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.302726 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "kube-api-access-l5nhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:06.303530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.303508 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:06.304871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.304845 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:06.305982 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.305957 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy" pod "30a1d1b2-c64b-4775-ac0c-115289b715d8" (UID: "30a1d1b2-c64b-4775-ac0c-115289b715d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:06.411607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.410797 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.411607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.410884 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.411607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.410901 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a1d1b2-c64b-4775-ac0c-115289b715d8-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.411607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.410914 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a1d1b2-c64b-4775-ac0c-115289b715d8-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.411607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.410930 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5nhq\" (UniqueName: \"kubernetes.io/projected/30a1d1b2-c64b-4775-ac0c-115289b715d8-kube-api-access-l5nhq\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.413029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.412663 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:06.753175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.753099 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" Oct 30 22:16:06.753175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.753099 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf" event={"ID":"f1f6e1fe-0ef9-4e24-a265-ea35721222da","Type":"ContainerDied","Data":"57affe14c12223465357a28b05c488d9ef7f6b85e23b68f1c49b4691a9766c43"} Oct 30 22:16:06.753175 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.753166 5508 scope.go:122] "RemoveContainer" containerID="7beab5db64d3e506929f25cffe6a97180312fcbdd37dc0d2d6ff943bd7788395" Oct 30 22:16:06.754715 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.754690 5508 generic.go:356] "Generic (PLEG): container finished" podID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerID="cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d" exitCode=0 Oct 30 22:16:06.754983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.754716 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" event={"ID":"30a1d1b2-c64b-4775-ac0c-115289b715d8","Type":"ContainerDied","Data":"cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d"} Oct 30 22:16:06.754983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.754758 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" Oct 30 22:16:06.754983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.754758 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44" event={"ID":"30a1d1b2-c64b-4775-ac0c-115289b715d8","Type":"ContainerDied","Data":"786633a1772991b5f366854d44e51a53f56129b70b8c6d36bbf6445d92e3c5c8"} Oct 30 22:16:06.763663 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.763644 5508 scope.go:122] "RemoveContainer" containerID="cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d" Oct 30 22:16:06.772951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.772930 5508 scope.go:122] "RemoveContainer" containerID="cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d" Oct 30 22:16:06.773225 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:06.773207 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d\": container with ID starting with cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d not found: ID does not exist" containerID="cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d" Oct 30 22:16:06.773278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.773231 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d"} err="failed to get container status \"cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d\": rpc error: code = NotFound desc = could not find container \"cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d\": container with ID starting with cabd01807cdd996990931c4771e17ca3ac98732cb736b33fd2897779afa6de7d not found: ID does not exist" Oct 30 22:16:06.918399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.918364 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44"] Oct 30 22:16:06.967539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:06.967504 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-qfb44"] Oct 30 22:16:07.031643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:07.031569 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf"] Oct 30 22:16:07.038003 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:07.037972 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d516c4-6cc7b5c4c9-zwrwf"] Oct 30 22:16:07.954293 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:07.954263 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" path="/var/lib/kubelet/pods/30a1d1b2-c64b-4775-ac0c-115289b715d8/volumes" Oct 30 22:16:07.954805 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:07.954762 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" path="/var/lib/kubelet/pods/f1f6e1fe-0ef9-4e24-a265-ea35721222da/volumes" Oct 30 22:16:08.445279 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:08.445234 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:08.620994 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:08.620960 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:10.310027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:10.309994 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8"] Oct 30 22:16:10.373117 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:10.373080 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:16:10.418737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:10.418707 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd"] Oct 30 22:16:10.706973 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:10.706943 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:10.907442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:10.907393 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:11.100465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:11.100422 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:11.324435 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:11.314920 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:12.468621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:12.468582 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:14.031565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:14.031486 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:14.180385 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:14.180349 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:15.310467 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:15.310418 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" podUID="56303faa-1a27-4e35-992e-99c463aa4829" containerName="payload-processing" containerID="cri-o://13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23" gracePeriod=25 Oct 30 22:16:15.419423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:15.419356 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerName="payload-pre-processing" containerID="cri-o://add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2" gracePeriod=25 Oct 30 22:16:15.895059 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:15.895024 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:17.493332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.493282 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:17.758555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.758471 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:17.941425 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.941390 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9"] Oct 30 22:16:17.942063 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942042 5508 cpu_manager.go:414] "containerMap: removing container" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942064 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942082 5508 cpu_manager.go:414] "containerMap: removing container" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942089 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942098 5508 cpu_manager.go:414] "containerMap: removing container" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942105 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942120 5508 cpu_manager.go:414] "containerMap: removing container" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerName="payload-processing" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942127 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerName="payload-processing" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942137 5508 cpu_manager.go:414] "containerMap: removing container" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942143 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942153 5508 cpu_manager.go:414] "containerMap: removing container" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942159 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942174 5508 cpu_manager.go:414] "containerMap: removing container" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942182 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942192 5508 cpu_manager.go:414] "containerMap: removing container" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" Oct 30 22:16:17.942180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942199 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942211 5508 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942219 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942229 5508 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942236 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942255 5508 cpu_manager.go:414] "containerMap: removing container" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942262 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942279 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942286 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942299 5508 cpu_manager.go:414] "containerMap: removing container" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942331 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942352 5508 cpu_manager.go:414] "containerMap: removing container" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942359 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942375 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942381 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942406 5508 cpu_manager.go:414] "containerMap: removing container" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942413 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942437 5508 cpu_manager.go:414] "containerMap: removing container" podUID="cf7694c5-5785-42c3-90a8-858c708f7904" containerName="maas-api" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942443 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7694c5-5785-42c3-90a8-858c708f7904" containerName="maas-api" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942610 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942624 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6e41a3-3330-4d4d-afb0-7c175c53f4d4" containerName="main" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942635 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942644 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942655 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942665 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942673 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942683 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942693 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942704 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942712 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a1d1b2-c64b-4775-ac0c-115289b715d8" containerName="istio-proxy" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942723 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7694c5-5785-42c3-90a8-858c708f7904" containerName="maas-api" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942737 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942747 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942757 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f6e1fe-0ef9-4e24-a265-ea35721222da" containerName="payload-processing" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942765 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:16:17.942951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.942776 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:16:17.946673 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.946650 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:17.950202 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.950176 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:17.999883 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:17.999847 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj2dz\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000078 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000048 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000163 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000110 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000234 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000177 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000234 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000226 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000289 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000347 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000496 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000438 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.000496 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.000472 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.016491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.016416 5508 generic.go:356] "Generic (PLEG): container finished" podID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerID="4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343" exitCode=0 Oct 30 22:16:18.016649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.016626 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" event={"ID":"8f9c240d-6eca-4f39-8f16-0ce7a40cf274","Type":"ContainerDied","Data":"4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343"} Oct 30 22:16:18.017153 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.017132 5508 scope.go:122] "RemoveContainer" containerID="4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343" Oct 30 22:16:18.019343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.019291 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.019444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.019388 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.019444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.019419 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.047086 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.047052 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9"] Oct 30 22:16:18.102258 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102224 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102268 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102292 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102382 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102455 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102526 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102703 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102649 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.102757 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.102703 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.103096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.103076 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.103207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.103125 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.103276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.103259 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj2dz\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.103349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.103293 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.103500 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.103458 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.103728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.103707 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.105249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.105119 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.105917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.105887 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.159670 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.159634 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.160709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.160683 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj2dz\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.230396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.230290 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" containerID="cri-o://31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7" gracePeriod=45 Oct 30 22:16:18.262873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.262844 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:18.280656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.280619 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" containerID="cri-o://3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404" gracePeriod=45 Oct 30 22:16:18.476595 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.473092 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:18.617977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.617941 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:16:18.622856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.622828 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:16:18.709160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.709131 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9"] Oct 30 22:16:18.710875 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:16:18.710837 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54eb8069_49be_41e8_a94c_26d8e51710aa.slice/crio-a67d894ed5ce249ad9718a34dccb9884e7957a14eedba127474a33806fe8a492 WatchSource:0}: Error finding container a67d894ed5ce249ad9718a34dccb9884e7957a14eedba127474a33806fe8a492: Status 404 returned error can't find the container with id a67d894ed5ce249ad9718a34dccb9884e7957a14eedba127474a33806fe8a492 Oct 30 22:16:18.710990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.710968 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs\" (UniqueName: \"kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs\") pod \"87c637bc-9f13-4e04-b988-a07c04ee163d\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " Oct 30 22:16:18.711054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711004 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home\") pod \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " Oct 30 22:16:18.711054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711029 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home\") pod \"87c637bc-9f13-4e04-b988-a07c04ee163d\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " Oct 30 22:16:18.711165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711075 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm\") pod \"87c637bc-9f13-4e04-b988-a07c04ee163d\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " Oct 30 22:16:18.711165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711100 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache\") pod \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " Oct 30 22:16:18.711272 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711249 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home" pod "bc09539f-7ac0-4241-b29a-cd19a9cd5f96" (UID: "bc09539f-7ac0-4241-b29a-cd19a9cd5f96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.711389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711361 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache" pod "bc09539f-7ac0-4241-b29a-cd19a9cd5f96" (UID: "bc09539f-7ac0-4241-b29a-cd19a9cd5f96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.711460 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711132 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb\" (UniqueName: \"kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb\") pod \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " Oct 30 22:16:18.711521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711483 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home" pod "87c637bc-9f13-4e04-b988-a07c04ee163d" (UID: "87c637bc-9f13-4e04-b988-a07c04ee163d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.711521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711508 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9\" (UniqueName: \"kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9\") pod \"87c637bc-9f13-4e04-b988-a07c04ee163d\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " Oct 30 22:16:18.711624 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711581 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir\") pod \"87c637bc-9f13-4e04-b988-a07c04ee163d\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " Oct 30 22:16:18.711682 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711639 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm\") pod \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " Oct 30 22:16:18.711682 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711672 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir\") pod \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " Oct 30 22:16:18.711837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.711729 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache\") pod \"87c637bc-9f13-4e04-b988-a07c04ee163d\" (UID: \"87c637bc-9f13-4e04-b988-a07c04ee163d\") " Oct 30 22:16:18.712430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.712161 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir" pod "bc09539f-7ac0-4241-b29a-cd19a9cd5f96" (UID: "bc09539f-7ac0-4241-b29a-cd19a9cd5f96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.712430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.712228 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache" pod "87c637bc-9f13-4e04-b988-a07c04ee163d" (UID: "87c637bc-9f13-4e04-b988-a07c04ee163d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.712430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.712262 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir" pod "87c637bc-9f13-4e04-b988-a07c04ee163d" (UID: "87c637bc-9f13-4e04-b988-a07c04ee163d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.712430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.712377 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs\") pod \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\" (UID: \"bc09539f-7ac0-4241-b29a-cd19a9cd5f96\") " Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.713386 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.713415 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.713430 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.713444 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.713459 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.713472 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.714496 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm" pod "87c637bc-9f13-4e04-b988-a07c04ee163d" (UID: "87c637bc-9f13-4e04-b988-a07c04ee163d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.714497 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb" pod "bc09539f-7ac0-4241-b29a-cd19a9cd5f96" (UID: "bc09539f-7ac0-4241-b29a-cd19a9cd5f96"). InnerVolumeSpecName "kube-api-access-cwvwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.714584 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9" pod "87c637bc-9f13-4e04-b988-a07c04ee163d" (UID: "87c637bc-9f13-4e04-b988-a07c04ee163d"). InnerVolumeSpecName "kube-api-access-zb4k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.714610 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs" pod "87c637bc-9f13-4e04-b988-a07c04ee163d" (UID: "87c637bc-9f13-4e04-b988-a07c04ee163d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.714978 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm" pod "bc09539f-7ac0-4241-b29a-cd19a9cd5f96" (UID: "bc09539f-7ac0-4241-b29a-cd19a9cd5f96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.715197 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.715258 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.715648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.715291 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.716325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.715891 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs" pod "bc09539f-7ac0-4241-b29a-cd19a9cd5f96" (UID: "bc09539f-7ac0-4241-b29a-cd19a9cd5f96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:18.810261 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.810230 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:16:18.810413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.810229 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" podUID="56303faa-1a27-4e35-992e-99c463aa4829" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:16:18.815029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.815002 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87c637bc-9f13-4e04-b988-a07c04ee163d-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.815133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.815031 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwvwb\" (UniqueName: \"kubernetes.io/projected/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-kube-api-access-cwvwb\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.815133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.815043 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4k9\" (UniqueName: \"kubernetes.io/projected/87c637bc-9f13-4e04-b988-a07c04ee163d-kube-api-access-zb4k9\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.815133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.815053 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.815133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.815062 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc09539f-7ac0-4241-b29a-cd19a9cd5f96-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.815133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:18.815071 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c637bc-9f13-4e04-b988-a07c04ee163d-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.029382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.029216 5508 generic.go:356] "Generic (PLEG): container finished" podID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerID="3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404" exitCode=0 Oct 30 22:16:19.029698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.029670 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" event={"ID":"bc09539f-7ac0-4241-b29a-cd19a9cd5f96","Type":"ContainerDied","Data":"3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404"} Oct 30 22:16:19.029835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.029821 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" event={"ID":"bc09539f-7ac0-4241-b29a-cd19a9cd5f96","Type":"ContainerDied","Data":"6afbc0b2b97aead7e64b7c594d3c766675406281ca03e59baf071617b1764c87"} Oct 30 22:16:19.032335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.029928 5508 scope.go:122] "RemoveContainer" containerID="3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404" Oct 30 22:16:19.032335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.030094 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz" Oct 30 22:16:19.055605 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.055577 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" event={"ID":"8f9c240d-6eca-4f39-8f16-0ce7a40cf274","Type":"ContainerStarted","Data":"5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4"} Oct 30 22:16:19.060075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.060039 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" event={"ID":"54eb8069-49be-41e8-a94c-26d8e51710aa","Type":"ContainerStarted","Data":"dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6"} Oct 30 22:16:19.060191 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.060099 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" event={"ID":"54eb8069-49be-41e8-a94c-26d8e51710aa","Type":"ContainerStarted","Data":"a67d894ed5ce249ad9718a34dccb9884e7957a14eedba127474a33806fe8a492"} Oct 30 22:16:19.062683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.062653 5508 generic.go:356] "Generic (PLEG): container finished" podID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerID="31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7" exitCode=0 Oct 30 22:16:19.062809 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.062708 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" Oct 30 22:16:19.062809 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.062711 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" event={"ID":"87c637bc-9f13-4e04-b988-a07c04ee163d","Type":"ContainerDied","Data":"31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7"} Oct 30 22:16:19.062809 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.062756 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl" event={"ID":"87c637bc-9f13-4e04-b988-a07c04ee163d","Type":"ContainerDied","Data":"2e445483fec7d27f0147bec8bfe8fe6b925761755cb5ad9313aec05dffe1c5f1"} Oct 30 22:16:19.069014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.068994 5508 scope.go:122] "RemoveContainer" containerID="3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404" Oct 30 22:16:19.069367 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:19.069339 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404\": container with ID starting with 3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404 not found: ID does not exist" containerID="3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404" Oct 30 22:16:19.069459 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.069375 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404"} err="failed to get container status \"3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404\": rpc error: code = NotFound desc = could not find container \"3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404\": container with ID starting with 3aa4b6c74576f5a06289f071c57e1cb3c708421ea2737a7187cf425b2c1b6404 not found: ID does not exist" Oct 30 22:16:19.069459 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.069398 5508 scope.go:122] "RemoveContainer" containerID="31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7" Oct 30 22:16:19.088944 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.087458 5508 scope.go:122] "RemoveContainer" containerID="31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7" Oct 30 22:16:19.088944 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:19.087956 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7\": container with ID starting with 31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7 not found: ID does not exist" containerID="31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7" Oct 30 22:16:19.088944 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.087994 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7"} err="failed to get container status \"31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7\": rpc error: code = NotFound desc = could not find container \"31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7\": container with ID starting with 31525ee438dea300983794c7e5d448bb39189028d7278bad2fe2f85b30a294d7 not found: ID does not exist" Oct 30 22:16:19.239755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.239720 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:16:19.241206 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.241176 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:19.241319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.241252 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:19.263043 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.263017 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:19.265027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.264999 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:19.265155 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.265048 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:19.328282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.328220 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podStartSLOduration=2.328202323 podStartE2EDuration="2.328202323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:19.211757825 +0000 UTC m=+1989.401295524" watchObservedRunningTime="2026-10-30 22:16:19.328202323 +0000 UTC m=+1989.517740002" Oct 30 22:16:19.329286 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.329255 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz"] Oct 30 22:16:19.332599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.332573 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-w4-b0c55f-kserve-5d8d6b976-hp6kz"] Oct 30 22:16:19.411289 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.411220 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:19.462701 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.462630 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl"] Oct 30 22:16:19.466198 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.466079 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/distinct-2-w4-b0c55f-kserve-7b46d47d99-fdgtl"] Oct 30 22:16:19.482557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.482446 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:16:19.575875 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.575740 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:19.955577 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.955545 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" path="/var/lib/kubelet/pods/87c637bc-9f13-4e04-b988-a07c04ee163d/volumes" Oct 30 22:16:19.956029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:19.956014 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" path="/var/lib/kubelet/pods/bc09539f-7ac0-4241-b29a-cd19a9cd5f96/volumes" Oct 30 22:16:20.238276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:20.238180 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:20.238276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:20.238234 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:20.263385 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:20.263356 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:20.263525 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:20.263414 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:21.237733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:21.237696 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.238210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:21.237751 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:21.263898 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:21.263871 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.264031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:21.263925 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:22.238588 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:22.238555 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.238967 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:22.238605 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:22.270341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:22.264186 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.270341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:22.264266 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:23.238656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:23.238619 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.239038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:23.238672 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:23.263766 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:23.263733 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.263919 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:23.263788 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:23.269116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:23.269057 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:24.237999 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:24.237962 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.238158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:24.238036 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 22:16:24.263401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:24.263366 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.263772 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:24.263428 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:24.670353 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:24.670281 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:24.858893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:24.858859 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:25.014145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.014059 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz"] Oct 30 22:16:25.014577 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.014546 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" containerID="cri-o://a5175a9b856c3d6cb6e43ee5ea9c4a8ca7040407fe82038cfd43164631c2eb20" gracePeriod=30 Oct 30 22:16:25.029761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.029723 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:25.250344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.250287 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:16:25.252189 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.252161 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:16:25.263702 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.263668 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:16:25.264208 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.263729 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 22:16:25.495862 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:25.495830 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:26.267886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:26.267858 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:26.268382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:26.268360 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:26.269098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:26.269078 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:16:27.967762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:27.963129 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:28.162224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:28.162179 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:28.810874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:28.810839 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:16:28.811042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:28.810849 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" podUID="56303faa-1a27-4e35-992e-99c463aa4829" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:16:29.746812 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:29.746779 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:29.966260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:29.966226 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:30.137040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.136966 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:30.231358 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.231327 5508 generic.go:356] "Generic (PLEG): container finished" podID="323cc524-025b-4d6d-bf45-845ac6156590" containerID="a5175a9b856c3d6cb6e43ee5ea9c4a8ca7040407fe82038cfd43164631c2eb20" exitCode=0 Oct 30 22:16:30.231519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.231395 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" event={"ID":"323cc524-025b-4d6d-bf45-845ac6156590","Type":"ContainerDied","Data":"a5175a9b856c3d6cb6e43ee5ea9c4a8ca7040407fe82038cfd43164631c2eb20"} Oct 30 22:16:30.231519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.231456 5508 scope.go:122] "RemoveContainer" containerID="892dbd9302762b49f303936b27f53144b060f158ddfd46ffdc2330828859d54b" Oct 30 22:16:30.439737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.439704 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8"] Oct 30 22:16:30.561063 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.561023 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:30.726560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.726535 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:16:30.833101 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833071 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833152 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833225 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833339 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833368 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833454 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833491 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833516 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.833572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833554 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.834232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833592 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.834232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833681 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo\") pod \"323cc524-025b-4d6d-bf45-845ac6156590\" (UID: \"323cc524-025b-4d6d-bf45-845ac6156590\") " Oct 30 22:16:30.834232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833834 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:30.834232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.834028 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:30.834486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.833790 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:30.834486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.834285 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.834486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.834331 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.834486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.834345 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cc524-025b-4d6d-bf45-845ac6156590-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.834486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.834359 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.835773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.835745 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:30.835905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.835785 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "kube-api-access-9kwc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:30.835905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.835816 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:30.836018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.836005 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo" pod "323cc524-025b-4d6d-bf45-845ac6156590" (UID: "323cc524-025b-4d6d-bf45-845ac6156590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:30.848567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.846256 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:30.935386 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.935346 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwc7\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-kube-api-access-9kwc7\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.935386 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.935380 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.935557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.935393 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cc524-025b-4d6d-bf45-845ac6156590-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.935557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.935402 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cc524-025b-4d6d-bf45-845ac6156590-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:30.935557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:30.935411 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cc524-025b-4d6d-bf45-845ac6156590-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.240138 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:31.240103 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" Oct 30 22:16:31.240341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:31.240107 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz" event={"ID":"323cc524-025b-4d6d-bf45-845ac6156590","Type":"ContainerDied","Data":"e885690c8025e59253f18923c9878a3f7f867c280b25c00c17b1da83cf9049be"} Oct 30 22:16:31.240341 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:31.240264 5508 scope.go:122] "RemoveContainer" containerID="a5175a9b856c3d6cb6e43ee5ea9c4a8ca7040407fe82038cfd43164631c2eb20" Oct 30 22:16:31.406925 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:31.406892 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz"] Oct 30 22:16:31.456109 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:31.456086 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7c826c-openshift-default-6678bbc987-52qfz"] Oct 30 22:16:31.954940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:31.954910 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323cc524-025b-4d6d-bf45-845ac6156590" path="/var/lib/kubelet/pods/323cc524-025b-4d6d-bf45-845ac6156590/volumes" Oct 30 22:16:32.489910 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:32.489879 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:32.888899 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:32.888856 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:35.124018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.123931 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:35.603079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.603052 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 22:16:35.687833 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.687780 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 22:16:35.714710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.714684 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l\" (UniqueName: \"kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l\") pod \"56303faa-1a27-4e35-992e-99c463aa4829\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " Oct 30 22:16:35.714854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.714777 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume\") pod \"56303faa-1a27-4e35-992e-99c463aa4829\" (UID: \"56303faa-1a27-4e35-992e-99c463aa4829\") " Oct 30 22:16:35.715131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.715109 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume" pod "56303faa-1a27-4e35-992e-99c463aa4829" (UID: "56303faa-1a27-4e35-992e-99c463aa4829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:35.715443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.715425 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56303faa-1a27-4e35-992e-99c463aa4829-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.717003 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.716984 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l" pod "56303faa-1a27-4e35-992e-99c463aa4829" (UID: "56303faa-1a27-4e35-992e-99c463aa4829"). InnerVolumeSpecName "kube-api-access-2x28l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:35.816829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.816797 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq\" (UniqueName: \"kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq\") pod \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " Oct 30 22:16:35.816987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.816957 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume\") pod \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\" (UID: \"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc\") " Oct 30 22:16:35.817424 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.817387 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume" pod "6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" (UID: "6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:35.817661 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.817637 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x28l\" (UniqueName: \"kubernetes.io/projected/56303faa-1a27-4e35-992e-99c463aa4829-kube-api-access-2x28l\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.817711 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.817669 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.819079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.819052 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq" pod "6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" (UID: "6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc"). InnerVolumeSpecName "kube-api-access-59mjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:35.919457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:35.919421 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59mjq\" (UniqueName: \"kubernetes.io/projected/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc-kube-api-access-59mjq\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:36.278155 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.278076 5508 generic.go:356] "Generic (PLEG): container finished" podID="56303faa-1a27-4e35-992e-99c463aa4829" containerID="13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23" exitCode=0 Oct 30 22:16:36.278155 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.278120 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" Oct 30 22:16:36.278727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.278164 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" event={"ID":"56303faa-1a27-4e35-992e-99c463aa4829","Type":"ContainerDied","Data":"13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23"} Oct 30 22:16:36.278727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.278231 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8" event={"ID":"56303faa-1a27-4e35-992e-99c463aa4829","Type":"ContainerDied","Data":"9b20c95c4a7fb916ce1c87bfe0d17cce2d1b4a750bdaea87af8c77c0124bc323"} Oct 30 22:16:36.278727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.278265 5508 scope.go:122] "RemoveContainer" containerID="13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23" Oct 30 22:16:36.279718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.279696 5508 generic.go:356] "Generic (PLEG): container finished" podID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerID="add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2" exitCode=0 Oct 30 22:16:36.279883 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.279743 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" Oct 30 22:16:36.279883 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.279770 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" event={"ID":"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc","Type":"ContainerDied","Data":"add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2"} Oct 30 22:16:36.279883 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.279809 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd" event={"ID":"6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc","Type":"ContainerDied","Data":"cf758bcc0f9ec03817fbcb1c2f98a44acb05680868a2fadb15aeb0b2e6079452"} Oct 30 22:16:36.288396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.288379 5508 scope.go:122] "RemoveContainer" containerID="13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23" Oct 30 22:16:36.288653 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:36.288635 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23\": container with ID starting with 13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23 not found: ID does not exist" containerID="13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23" Oct 30 22:16:36.288709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.288662 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23"} err="failed to get container status \"13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23\": rpc error: code = NotFound desc = could not find container \"13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23\": container with ID starting with 13e18c8f002adf064a3aea8972fda4e3c699c5945d4b03ea6b59e10103e39f23 not found: ID does not exist" Oct 30 22:16:36.288709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.288679 5508 scope.go:122] "RemoveContainer" containerID="add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2" Oct 30 22:16:36.299881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.299862 5508 scope.go:122] "RemoveContainer" containerID="add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2" Oct 30 22:16:36.300450 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:36.300426 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2\": container with ID starting with add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2 not found: ID does not exist" containerID="add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2" Oct 30 22:16:36.300535 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.300456 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2"} err="failed to get container status \"add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2\": rpc error: code = NotFound desc = could not find container \"add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2\": container with ID starting with add441b638f49d2e36cedfd3ea7f746bf88726e376fd04fb9181c9e1301805c2 not found: ID does not exist" Oct 30 22:16:36.440622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.440593 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd"] Oct 30 22:16:36.443495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.443471 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b0c55f-6ccddfb9c5-zx6wd"] Oct 30 22:16:36.552402 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.552297 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8"] Oct 30 22:16:36.556455 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:36.556429 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b0c55f-64cff785-mcvb8"] Oct 30 22:16:37.445089 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:37.445054 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:37.610924 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:37.610892 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:37.777885 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:37.777788 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:38.016075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:38.013606 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:38.054719 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:38.054603 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56303faa-1a27-4e35-992e-99c463aa4829" path="/var/lib/kubelet/pods/56303faa-1a27-4e35-992e-99c463aa4829/volumes" Oct 30 22:16:38.055464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:38.055444 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" path="/var/lib/kubelet/pods/6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc/volumes" Oct 30 22:16:38.174412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:38.174376 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:39.953254 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:39.953219 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:40.121584 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:40.121547 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:40.381195 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:40.381155 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:16:40.411902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:40.411871 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:41.451896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:41.451865 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:44.102819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:44.102790 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:44.264463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:44.264352 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:44.802047 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:44.802014 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:45.361962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.361747 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6"] Oct 30 22:16:45.362804 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.362771 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" containerID="cri-o://02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a" gracePeriod=30 Oct 30 22:16:45.398473 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.398439 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5"] Oct 30 22:16:45.398694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.398671 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" containerID="cri-o://dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e" gracePeriod=30 Oct 30 22:16:45.462507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.462439 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" containerID="cri-o://ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f" gracePeriod=45 Oct 30 22:16:45.517338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.507015 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:45.517338 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.507080 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:45.602727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.602682 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:45.849442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.849411 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:16:45.905057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905031 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home\") pod \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " Oct 30 22:16:45.905222 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905074 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs\") pod \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " Oct 30 22:16:45.905222 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905103 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache\") pod \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " Oct 30 22:16:45.905377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905263 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home" pod "3fb090b2-8a3c-49a3-8c06-13f907d41fa2" (UID: "3fb090b2-8a3c-49a3-8c06-13f907d41fa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.905377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905283 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7\" (UniqueName: \"kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7\") pod \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " Oct 30 22:16:45.905493 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905425 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir\") pod \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " Oct 30 22:16:45.905493 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905453 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache" pod "3fb090b2-8a3c-49a3-8c06-13f907d41fa2" (UID: "3fb090b2-8a3c-49a3-8c06-13f907d41fa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.905586 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905535 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm\") pod \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\" (UID: \"3fb090b2-8a3c-49a3-8c06-13f907d41fa2\") " Oct 30 22:16:45.905708 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.905689 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir" pod "3fb090b2-8a3c-49a3-8c06-13f907d41fa2" (UID: "3fb090b2-8a3c-49a3-8c06-13f907d41fa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.906336 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.906279 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.906336 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.906329 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.906444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.906342 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.907737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.907712 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm" pod "3fb090b2-8a3c-49a3-8c06-13f907d41fa2" (UID: "3fb090b2-8a3c-49a3-8c06-13f907d41fa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.907832 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.907718 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs" pod "3fb090b2-8a3c-49a3-8c06-13f907d41fa2" (UID: "3fb090b2-8a3c-49a3-8c06-13f907d41fa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:45.907832 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:45.907746 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7" pod "3fb090b2-8a3c-49a3-8c06-13f907d41fa2" (UID: "3fb090b2-8a3c-49a3-8c06-13f907d41fa2"). InnerVolumeSpecName "kube-api-access-cvxw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:46.008384 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.008285 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxw7\" (UniqueName: \"kubernetes.io/projected/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-kube-api-access-cvxw7\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.008384 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.008338 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.008384 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.008353 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb090b2-8a3c-49a3-8c06-13f907d41fa2-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.383610 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.383577 5508 generic.go:356] "Generic (PLEG): container finished" podID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerID="ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f" exitCode=0 Oct 30 22:16:46.384051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.383627 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" Oct 30 22:16:46.384051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.383652 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" event={"ID":"3fb090b2-8a3c-49a3-8c06-13f907d41fa2","Type":"ContainerDied","Data":"ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f"} Oct 30 22:16:46.384051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.383693 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8" event={"ID":"3fb090b2-8a3c-49a3-8c06-13f907d41fa2","Type":"ContainerDied","Data":"013f83ad57a7c3a3aa109fd74aface10df3388a3854cc133c723e821277f1217"} Oct 30 22:16:46.384051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.383710 5508 scope.go:122] "RemoveContainer" containerID="ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f" Oct 30 22:16:46.393865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.393843 5508 scope.go:122] "RemoveContainer" containerID="ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f" Oct 30 22:16:46.394131 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:46.394113 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f\": container with ID starting with ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f not found: ID does not exist" containerID="ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f" Oct 30 22:16:46.394194 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.394139 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f"} err="failed to get container status \"ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f\": rpc error: code = NotFound desc = could not find container \"ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f\": container with ID starting with ae3af27b385f22d90b96acbbc5e3d850d5322200627ccb1e8a3c32f76993065f not found: ID does not exist" Oct 30 22:16:46.545705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.545666 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8"] Oct 30 22:16:46.553166 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:46.553138 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/distinct-w2-0c99b1-kserve-74b7bd557c-w78b8"] Oct 30 22:16:47.060625 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:47.060588 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k"] Oct 30 22:16:47.060835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:47.060798 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" containerID="cri-o://5a26fe38e243111723223be6c27c4c3a25c065823884f63633092eadf4957a72" gracePeriod=30 Oct 30 22:16:47.109769 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:47.109735 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p"] Oct 30 22:16:47.110008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:47.109974 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" containerID="cri-o://92c6e0b1a6a3e8ccbfe1db4d0f07da6f8072de0c21646f349c0c62560f699f33" gracePeriod=30 Oct 30 22:16:47.953224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:47.953183 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" path="/var/lib/kubelet/pods/3fb090b2-8a3c-49a3-8c06-13f907d41fa2/volumes" Oct 30 22:16:48.605837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:48.605806 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:49.708620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:49.708581 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:49.708989 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:49.708632 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:50.466250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.466212 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:50.647287 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.647207 5508 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:16:50.647287 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.647274 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:16:50.693494 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.693467 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 22:16:50.742958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.742929 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:16:50.769313 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769277 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769427 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769322 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769457 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769541 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769516 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.769578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769554 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769612 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769584 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769626 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769679 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769654 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769702 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769776 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769738 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy\") pod \"cce63103-453c-4612-b6dd-ae756c80acb2\" (UID: \"cce63103-453c-4612-b6dd-ae756c80acb2\") " Oct 30 22:16:50.769871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769712 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.769871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769852 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.770006 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.769776 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.770106 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.770077 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:50.770571 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.770544 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce63103-453c-4612-b6dd-ae756c80acb2-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.770729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.770576 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.770729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.770592 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.770729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.770607 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.770729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.770622 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.771961 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.771925 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.772058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.771976 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:50.772058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.772003 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:50.772177 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.772155 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd" pod "cce63103-453c-4612-b6dd-ae756c80acb2" (UID: "cce63103-453c-4612-b6dd-ae756c80acb2"). InnerVolumeSpecName "kube-api-access-r2mwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:50.867566 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.867538 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x"] Oct 30 22:16:50.871604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871581 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871638 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871655 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871686 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871731 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871803 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871866 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.871945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871890 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.871945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871920 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.872136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.871951 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn\") pod \"bda608a1-3b58-445c-b3c7-9e75d4044544\" (UID: \"bda608a1-3b58-445c-b3c7-9e75d4044544\") " Oct 30 22:16:50.872136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872125 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.872203 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872165 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:50.872426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872338 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.872426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872418 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.872729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872689 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872706 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872720 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872735 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872744 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda608a1-3b58-445c-b3c7-9e75d4044544-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872768 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872779 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce63103-453c-4612-b6dd-ae756c80acb2-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872788 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2mwd\" (UniqueName: \"kubernetes.io/projected/cce63103-453c-4612-b6dd-ae756c80acb2-kube-api-access-r2mwd\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.872892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.872797 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce63103-453c-4612-b6dd-ae756c80acb2-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.874248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.874225 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:50.874395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.874341 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "kube-api-access-wgkhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:50.874527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.874501 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:50.874621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.874597 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token" pod "bda608a1-3b58-445c-b3c7-9e75d4044544" (UID: "bda608a1-3b58-445c-b3c7-9e75d4044544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:50.974005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.973937 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.974005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.973966 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.974005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.973976 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkhn\" (UniqueName: \"kubernetes.io/projected/bda608a1-3b58-445c-b3c7-9e75d4044544-kube-api-access-wgkhn\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:50.974005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:50.973985 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda608a1-3b58-445c-b3c7-9e75d4044544-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.236462 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.236356 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:51.421194 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.421157 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:51.422731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.422704 5508 generic.go:356] "Generic (PLEG): container finished" podID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerID="dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e" exitCode=0 Oct 30 22:16:51.422881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.422778 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" Oct 30 22:16:51.422881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.422789 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" event={"ID":"bda608a1-3b58-445c-b3c7-9e75d4044544","Type":"ContainerDied","Data":"dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e"} Oct 30 22:16:51.422881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.422869 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5" event={"ID":"bda608a1-3b58-445c-b3c7-9e75d4044544","Type":"ContainerDied","Data":"95e30111338e5f3f4080085356f97914979a11c96026753433b36c8e385e3192"} Oct 30 22:16:51.423045 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.422901 5508 scope.go:122] "RemoveContainer" containerID="dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e" Oct 30 22:16:51.425007 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.424981 5508 generic.go:356] "Generic (PLEG): container finished" podID="cce63103-453c-4612-b6dd-ae756c80acb2" containerID="02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a" exitCode=0 Oct 30 22:16:51.425132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.425022 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" event={"ID":"cce63103-453c-4612-b6dd-ae756c80acb2","Type":"ContainerDied","Data":"02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a"} Oct 30 22:16:51.425132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.425083 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" event={"ID":"cce63103-453c-4612-b6dd-ae756c80acb2","Type":"ContainerDied","Data":"ff50c2910eaf37a656f601b22a57d625282b685f5bca17bebc4de88d18115fe0"} Oct 30 22:16:51.425287 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.425270 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6" Oct 30 22:16:51.437740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.437687 5508 scope.go:122] "RemoveContainer" containerID="dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e" Oct 30 22:16:51.438097 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:51.438071 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e\": container with ID starting with dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e not found: ID does not exist" containerID="dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e" Oct 30 22:16:51.438181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.438110 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e"} err="failed to get container status \"dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e\": rpc error: code = NotFound desc = could not find container \"dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e\": container with ID starting with dd406105066f435c549671e173fd5bc6d7c3225651e9b47bf87c02afd059246e not found: ID does not exist" Oct 30 22:16:51.438181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.438135 5508 scope.go:122] "RemoveContainer" containerID="02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a" Oct 30 22:16:51.489209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.488034 5508 scope.go:122] "RemoveContainer" containerID="02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a" Oct 30 22:16:51.489209 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:51.488460 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a\": container with ID starting with 02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a not found: ID does not exist" containerID="02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a" Oct 30 22:16:51.489209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.488499 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a"} err="failed to get container status \"02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a\": rpc error: code = NotFound desc = could not find container \"02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a\": container with ID starting with 02418e1f4f98b18b0c86c1787cd3e9b352b175bb7f3ba6953a5dcfb56b0a272a not found: ID does not exist" Oct 30 22:16:51.588840 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.588806 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6"] Oct 30 22:16:51.593800 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.593759 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-ff9h6"] Oct 30 22:16:51.705846 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.705814 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5"] Oct 30 22:16:51.709959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.709934 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-4kfh5"] Oct 30 22:16:51.953905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.953868 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" path="/var/lib/kubelet/pods/bda608a1-3b58-445c-b3c7-9e75d4044544/volumes" Oct 30 22:16:51.954392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:51.954377 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" path="/var/lib/kubelet/pods/cce63103-453c-4612-b6dd-ae756c80acb2/volumes" Oct 30 22:16:52.451133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.451095 5508 generic.go:356] "Generic (PLEG): container finished" podID="a53062e4-bdde-401c-a512-465b522fcd64" containerID="92c6e0b1a6a3e8ccbfe1db4d0f07da6f8072de0c21646f349c0c62560f699f33" exitCode=0 Oct 30 22:16:52.451381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.451216 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" event={"ID":"a53062e4-bdde-401c-a512-465b522fcd64","Type":"ContainerDied","Data":"92c6e0b1a6a3e8ccbfe1db4d0f07da6f8072de0c21646f349c0c62560f699f33"} Oct 30 22:16:52.453141 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.453112 5508 generic.go:356] "Generic (PLEG): container finished" podID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerID="5a26fe38e243111723223be6c27c4c3a25c065823884f63633092eadf4957a72" exitCode=0 Oct 30 22:16:52.453262 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.453221 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" event={"ID":"42f08e44-69fa-4a74-9f97-5ecaf6606b55","Type":"ContainerDied","Data":"5a26fe38e243111723223be6c27c4c3a25c065823884f63633092eadf4957a72"} Oct 30 22:16:52.531515 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.531489 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 22:16:52.534893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.534874 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:16:52.626103 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626067 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.626267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626115 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.626267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626143 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.626267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626185 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.626267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626207 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.626267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626225 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.626578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626416 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.626578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626462 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.626578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626488 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.626578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626509 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.626578 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626546 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.626824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626600 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.626824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626643 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.626824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626670 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:52.626824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626711 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.626824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626778 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.626824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626811 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.627164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626852 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.627164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626904 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.627164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626957 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.627164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.626987 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.627164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.627071 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo\") pod \"a53062e4-bdde-401c-a512-465b522fcd64\" (UID: \"a53062e4-bdde-401c-a512-465b522fcd64\") " Oct 30 22:16:52.627164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.627030 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.627515 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.627130 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy\") pod \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\" (UID: \"42f08e44-69fa-4a74-9f97-5ecaf6606b55\") " Oct 30 22:16:52.627515 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.627481 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.627763 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.627702 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.628020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.627994 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628029 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628051 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628066 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628078 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628092 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628104 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628120 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.628327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628006 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:52.628547 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.628344 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.629379 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.629349 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "kube-api-access-q6zfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:52.629536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.629436 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:52.629841 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.629797 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:52.629952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.629883 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.630283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.630255 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "kube-api-access-657gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:52.630571 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.630553 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:52.630631 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.630566 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy" pod "42f08e44-69fa-4a74-9f97-5ecaf6606b55" (UID: "42f08e44-69fa-4a74-9f97-5ecaf6606b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:52.630710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.630695 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo" pod "a53062e4-bdde-401c-a512-465b522fcd64" (UID: "a53062e4-bdde-401c-a512-465b522fcd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.729887 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.729924 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.729938 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53062e4-bdde-401c-a512-465b522fcd64-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.729954 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53062e4-bdde-401c-a512-465b522fcd64-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.729969 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53062e4-bdde-401c-a512-465b522fcd64-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.729986 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.730001 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zfp\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-kube-api-access-q6zfp\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.730018 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657gr\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-kube-api-access-657gr\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.730035 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f08e44-69fa-4a74-9f97-5ecaf6606b55-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:52.735145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:52.730048 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53062e4-bdde-401c-a512-465b522fcd64-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.461479 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.461443 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" Oct 30 22:16:53.461479 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.461463 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p" event={"ID":"a53062e4-bdde-401c-a512-465b522fcd64","Type":"ContainerDied","Data":"82d58aa29b3e92711bc71976ae3b1d46bdfc5820fceae23b158f7127e7844e88"} Oct 30 22:16:53.461976 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.461553 5508 scope.go:122] "RemoveContainer" containerID="92c6e0b1a6a3e8ccbfe1db4d0f07da6f8072de0c21646f349c0c62560f699f33" Oct 30 22:16:53.463253 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.463223 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" event={"ID":"42f08e44-69fa-4a74-9f97-5ecaf6606b55","Type":"ContainerDied","Data":"529f30c20a56c8ba4f4cb6602fccfe2b8ba09c036c4659a54cef62606468b91c"} Oct 30 22:16:53.463482 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.463256 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k" Oct 30 22:16:53.472972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.472953 5508 scope.go:122] "RemoveContainer" containerID="5a26fe38e243111723223be6c27c4c3a25c065823884f63633092eadf4957a72" Oct 30 22:16:53.677098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.677067 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k"] Oct 30 22:16:53.725392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.725326 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-qjd9k"] Oct 30 22:16:53.787142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.787111 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p"] Oct 30 22:16:53.791165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.791142 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-cj48p"] Oct 30 22:16:53.953811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.953771 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" path="/var/lib/kubelet/pods/42f08e44-69fa-4a74-9f97-5ecaf6606b55/volumes" Oct 30 22:16:53.954278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:53.954263 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53062e4-bdde-401c-a512-465b522fcd64" path="/var/lib/kubelet/pods/a53062e4-bdde-401c-a512-465b522fcd64/volumes" Oct 30 22:16:54.801453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:54.801419 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:55.868536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:55.868499 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" podUID="fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" containerName="payload-processing" containerID="cri-o://ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12" gracePeriod=25 Oct 30 22:16:56.150846 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.150821 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:16:56.293064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.293033 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume\") pod \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " Oct 30 22:16:56.293266 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.293160 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8\" (UniqueName: \"kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8\") pod \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\" (UID: \"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d\") " Oct 30 22:16:56.293527 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.293498 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume" pod "fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" (UID: "fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:56.295616 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.295592 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8" pod "fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" (UID: "fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d"). InnerVolumeSpecName "kube-api-access-7vdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:56.339952 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.339923 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:56.394346 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.394268 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.394346 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.394293 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vdc8\" (UniqueName: \"kubernetes.io/projected/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d-kube-api-access-7vdc8\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:56.487900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.487868 5508 generic.go:356] "Generic (PLEG): container finished" podID="fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" containerID="ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12" exitCode=0 Oct 30 22:16:56.488075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.487911 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" Oct 30 22:16:56.488075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.487940 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" event={"ID":"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d","Type":"ContainerDied","Data":"ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12"} Oct 30 22:16:56.488075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.487994 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x" event={"ID":"fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d","Type":"ContainerDied","Data":"565276c48730636d66967397bfb6da18d8ae1d049b43e210401e3dcb78792178"} Oct 30 22:16:56.488075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.488023 5508 scope.go:122] "RemoveContainer" containerID="ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12" Oct 30 22:16:56.498461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.498440 5508 scope.go:122] "RemoveContainer" containerID="ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12" Oct 30 22:16:56.498735 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:16:56.498712 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12\": container with ID starting with ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12 not found: ID does not exist" containerID="ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12" Oct 30 22:16:56.498811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.498746 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12"} err="failed to get container status \"ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12\": rpc error: code = NotFound desc = could not find container \"ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12\": container with ID starting with ae33bdf1afc88d3c7c76bbf8fabb32e926b46e614e7c7a5930c38db663fa9d12 not found: ID does not exist" Oct 30 22:16:56.564640 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.564614 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:56.651601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.651532 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x"] Oct 30 22:16:56.681977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.681933 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-0a7679-6b45f9bc9f-9hj4x"] Oct 30 22:16:56.727970 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:56.727931 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:57.338266 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:57.338217 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:57.954798 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:16:57.954766 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" path="/var/lib/kubelet/pods/fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d/volumes" Oct 30 22:17:01.086523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:01.086493 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:02.805935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:02.805887 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.077160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.077115 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.249951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.249913 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.541242 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.541141 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.714720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.714681 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.862886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.862856 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn"] Oct 30 22:17:03.863349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.863046 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" containerID="cri-o://906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb" gracePeriod=30 Oct 30 22:17:03.883816 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:03.883782 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:07.261185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:07.261152 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:08.027996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:08.027964 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.036170 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.036131 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.207418 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.205459 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.241328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.241289 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 22:17:09.334332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334282 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334442 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334514 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334492 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334632 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334543 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334632 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334597 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334686 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334723 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334838 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334753 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334838 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334790 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert\") pod \"89cb7485-ccdd-4c73-81df-45d1511c986b\" (UID: \"89cb7485-ccdd-4c73-81df-45d1511c986b\") " Oct 30 22:17:09.334942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334878 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.334942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.334887 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.335174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.335109 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.335392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.335373 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.335485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.335392 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.335485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.335403 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.335485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.335349 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:09.335646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.335505 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.336731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.336708 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:09.337040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.337021 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.337112 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.337058 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "kube-api-access-qtpzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.337353 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.337333 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token" pod "89cb7485-ccdd-4c73-81df-45d1511c986b" (UID: "89cb7485-ccdd-4c73-81df-45d1511c986b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.386151 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.386121 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.436420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.436393 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.436420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.436419 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtpzb\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-kube-api-access-qtpzb\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.436598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.436433 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.436598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.436447 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cb7485-ccdd-4c73-81df-45d1511c986b-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.436598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.436459 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cb7485-ccdd-4c73-81df-45d1511c986b-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.436598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.436470 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cb7485-ccdd-4c73-81df-45d1511c986b-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.586432 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.586399 5508 generic.go:356] "Generic (PLEG): container finished" podID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerID="906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb" exitCode=0 Oct 30 22:17:09.586630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.586452 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" Oct 30 22:17:09.586630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.586465 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" event={"ID":"89cb7485-ccdd-4c73-81df-45d1511c986b","Type":"ContainerDied","Data":"906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb"} Oct 30 22:17:09.586630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.586521 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn" event={"ID":"89cb7485-ccdd-4c73-81df-45d1511c986b","Type":"ContainerDied","Data":"037b28db343dbd42d33ffccb624388a88032c747d6a92ba991e618ebcc5751ee"} Oct 30 22:17:09.586630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.586551 5508 scope.go:122] "RemoveContainer" containerID="906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb" Oct 30 22:17:09.598928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.598901 5508 scope.go:122] "RemoveContainer" containerID="906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb" Oct 30 22:17:09.599280 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:09.599216 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb\": container with ID starting with 906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb not found: ID does not exist" containerID="906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb" Oct 30 22:17:09.599280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.599252 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb"} err="failed to get container status \"906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb\": rpc error: code = NotFound desc = could not find container \"906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb\": container with ID starting with 906edbab992375eba4172610d8e0707f779fa586c567be22209ac1f8bf59fcbb not found: ID does not exist" Oct 30 22:17:09.767161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.767124 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.777201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.777174 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn"] Oct 30 22:17:09.788668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.788637 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-zv6fn"] Oct 30 22:17:09.892789 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.892711 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484"] Oct 30 22:17:09.952932 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.952903 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" path="/var/lib/kubelet/pods/89cb7485-ccdd-4c73-81df-45d1511c986b/volumes" Oct 30 22:17:09.999567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:09.999535 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:10.183012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:10.182926 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:10.381153 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:10.379037 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:10.454270 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:10.454181 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:17:10.546585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:10.546554 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:10.718362 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:10.718264 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:11.114087 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:11.114055 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:11.387439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:11.387348 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:11.660475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:11.660395 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:12.047588 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:12.047508 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:12.225144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:12.225107 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:12.393800 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:12.393769 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:13.937698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:13.937661 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:14.112484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:14.112449 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:14.501662 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:14.501603 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:14.692255 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:14.692220 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:14.893040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:14.892999 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" podUID="bf5d5ef8-df27-4f0a-bb33-2a64f409480a" containerName="payload-processing" containerID="cri-o://1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea" gracePeriod=25 Oct 30 22:17:15.250667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.250642 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:17:15.350749 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.350714 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v\" (UniqueName: \"kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v\") pod \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " Oct 30 22:17:15.350947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.350776 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume\") pod \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\" (UID: \"bf5d5ef8-df27-4f0a-bb33-2a64f409480a\") " Oct 30 22:17:15.351149 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.351125 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume" pod "bf5d5ef8-df27-4f0a-bb33-2a64f409480a" (UID: "bf5d5ef8-df27-4f0a-bb33-2a64f409480a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:15.351361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.351346 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.353148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.353121 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v" pod "bf5d5ef8-df27-4f0a-bb33-2a64f409480a" (UID: "bf5d5ef8-df27-4f0a-bb33-2a64f409480a"). InnerVolumeSpecName "kube-api-access-czr9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:15.452637 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.452599 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czr9v\" (UniqueName: \"kubernetes.io/projected/bf5d5ef8-df27-4f0a-bb33-2a64f409480a-kube-api-access-czr9v\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:15.673893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.673858 5508 generic.go:356] "Generic (PLEG): container finished" podID="bf5d5ef8-df27-4f0a-bb33-2a64f409480a" containerID="1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea" exitCode=0 Oct 30 22:17:15.674100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.673916 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" Oct 30 22:17:15.674100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.673916 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" event={"ID":"bf5d5ef8-df27-4f0a-bb33-2a64f409480a","Type":"ContainerDied","Data":"1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea"} Oct 30 22:17:15.674100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.673990 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484" event={"ID":"bf5d5ef8-df27-4f0a-bb33-2a64f409480a","Type":"ContainerDied","Data":"16da40ad8a752442817e3d919ac93df49cb9eaa5a1008e11261fa4034d47a68d"} Oct 30 22:17:15.674100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.674021 5508 scope.go:122] "RemoveContainer" containerID="1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea" Oct 30 22:17:15.684486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.684468 5508 scope.go:122] "RemoveContainer" containerID="1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea" Oct 30 22:17:15.684754 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:15.684735 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea\": container with ID starting with 1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea not found: ID does not exist" containerID="1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea" Oct 30 22:17:15.684828 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.684759 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea"} err="failed to get container status \"1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea\": rpc error: code = NotFound desc = could not find container \"1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea\": container with ID starting with 1d7767af3339dd447a03035c9bf333af696a4c6748c2eb0487404a70ddf5b9ea not found: ID does not exist" Oct 30 22:17:15.837523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.837495 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484"] Oct 30 22:17:15.886679 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.886654 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-e431fc-d48695487-t6484"] Oct 30 22:17:15.954078 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:15.954015 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf5d5ef8-df27-4f0a-bb33-2a64f409480a" path="/var/lib/kubelet/pods/bf5d5ef8-df27-4f0a-bb33-2a64f409480a/volumes" Oct 30 22:17:16.189646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:16.189612 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:17.252484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:17.252455 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:17.762062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:17.762028 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:17.988772 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:17.988734 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:18.164792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:18.164747 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:18.503440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:18.503344 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:18.702919 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:18.702888 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:18.889498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:18.889466 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:19.251754 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:19.251657 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:19.454565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:19.454522 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:19.797190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:19.797116 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:23.063608 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:23.063565 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:23.232686 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:23.232651 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:23.404118 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:23.404079 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:24.627812 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:24.627769 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:24.789148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:24.789112 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:24.996176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:24.996065 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:28.310543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:28.310505 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:31.299044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:31.299001 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28"] Oct 30 22:17:31.400271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:31.400237 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7"] Oct 30 22:17:31.665991 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:31.665958 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:31.820381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:31.820345 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:32.612572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:32.612540 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:32.832571 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:32.832534 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:34.100205 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:34.100175 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:34.360568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:34.360490 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:34.513043 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:34.513006 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:35.087544 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:35.087512 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:36.300422 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.300382 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" podUID="bac5c48b-8768-4e01-81c2-684c76dab0da" containerName="payload-processing" containerID="cri-o://6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c" gracePeriod=25 Oct 30 22:17:36.401076 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.401042 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" podUID="81b86dbf-0e5b-4c51-bb04-0a6292f15659" containerName="payload-pre-processing" containerID="cri-o://fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4" gracePeriod=25 Oct 30 22:17:36.577611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.577584 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:17:36.671691 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.671664 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:17:36.685927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.685902 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume\") pod \"bac5c48b-8768-4e01-81c2-684c76dab0da\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " Oct 30 22:17:36.686091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.685993 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp\" (UniqueName: \"kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp\") pod \"bac5c48b-8768-4e01-81c2-684c76dab0da\" (UID: \"bac5c48b-8768-4e01-81c2-684c76dab0da\") " Oct 30 22:17:36.686349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.686292 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume" pod "bac5c48b-8768-4e01-81c2-684c76dab0da" (UID: "bac5c48b-8768-4e01-81c2-684c76dab0da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:36.686586 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.686572 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac5c48b-8768-4e01-81c2-684c76dab0da-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.688261 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.688234 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp" pod "bac5c48b-8768-4e01-81c2-684c76dab0da" (UID: "bac5c48b-8768-4e01-81c2-684c76dab0da"). InnerVolumeSpecName "kube-api-access-75qkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:36.787627 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.787597 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp\" (UniqueName: \"kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp\") pod \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " Oct 30 22:17:36.787825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.787715 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume\") pod \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\" (UID: \"81b86dbf-0e5b-4c51-bb04-0a6292f15659\") " Oct 30 22:17:36.788130 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.788106 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume" pod "81b86dbf-0e5b-4c51-bb04-0a6292f15659" (UID: "81b86dbf-0e5b-4c51-bb04-0a6292f15659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:36.788130 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.788119 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qkp\" (UniqueName: \"kubernetes.io/projected/bac5c48b-8768-4e01-81c2-684c76dab0da-kube-api-access-75qkp\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.789888 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.789866 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp" pod "81b86dbf-0e5b-4c51-bb04-0a6292f15659" (UID: "81b86dbf-0e5b-4c51-bb04-0a6292f15659"). InnerVolumeSpecName "kube-api-access-2c6tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:36.894416 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.894378 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b86dbf-0e5b-4c51-bb04-0a6292f15659-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.894696 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.894677 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6tp\" (UniqueName: \"kubernetes.io/projected/81b86dbf-0e5b-4c51-bb04-0a6292f15659-kube-api-access-2c6tp\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:36.902825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.902785 5508 generic.go:356] "Generic (PLEG): container finished" podID="81b86dbf-0e5b-4c51-bb04-0a6292f15659" containerID="fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4" exitCode=0 Oct 30 22:17:36.903096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.902836 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" Oct 30 22:17:36.903096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.902866 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" event={"ID":"81b86dbf-0e5b-4c51-bb04-0a6292f15659","Type":"ContainerDied","Data":"fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4"} Oct 30 22:17:36.903096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.902959 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7" event={"ID":"81b86dbf-0e5b-4c51-bb04-0a6292f15659","Type":"ContainerDied","Data":"f1d3eb167dbd4d9aa84fc5548826089caad5a210765459abc15b6be00be08117"} Oct 30 22:17:36.903096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.902987 5508 scope.go:122] "RemoveContainer" containerID="fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4" Oct 30 22:17:36.907764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.907728 5508 generic.go:356] "Generic (PLEG): container finished" podID="bac5c48b-8768-4e01-81c2-684c76dab0da" containerID="6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c" exitCode=0 Oct 30 22:17:36.907880 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.907779 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" event={"ID":"bac5c48b-8768-4e01-81c2-684c76dab0da","Type":"ContainerDied","Data":"6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c"} Oct 30 22:17:36.907880 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.907832 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" event={"ID":"bac5c48b-8768-4e01-81c2-684c76dab0da","Type":"ContainerDied","Data":"53a444634b1da087d609b3370eebab7337f6568ad8a6cb26adfbb8fe38eb0ea0"} Oct 30 22:17:36.908020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.907922 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28" Oct 30 22:17:36.919140 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.919123 5508 scope.go:122] "RemoveContainer" containerID="fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4" Oct 30 22:17:36.919427 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:36.919405 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4\": container with ID starting with fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4 not found: ID does not exist" containerID="fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4" Oct 30 22:17:36.919524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.919440 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4"} err="failed to get container status \"fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4\": rpc error: code = NotFound desc = could not find container \"fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4\": container with ID starting with fbb12d8bc744896402010886053aaa997e90c3b811a32c8a01521c56d82fb3e4 not found: ID does not exist" Oct 30 22:17:36.919524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.919465 5508 scope.go:122] "RemoveContainer" containerID="6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c" Oct 30 22:17:36.930523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.930509 5508 scope.go:122] "RemoveContainer" containerID="6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c" Oct 30 22:17:36.930795 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:36.930766 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c\": container with ID starting with 6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c not found: ID does not exist" containerID="6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c" Oct 30 22:17:36.930856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:36.930789 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c"} err="failed to get container status \"6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c\": rpc error: code = NotFound desc = could not find container \"6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c\": container with ID starting with 6a2015f007e29c62e17321de2fb536dd37a4cc3d3cd52fe6bdee28981b079b2c not found: ID does not exist" Oct 30 22:17:37.087340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.087104 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28"] Oct 30 22:17:37.092742 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.092714 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-db492e-97dff7fc9-5zg28"] Oct 30 22:17:37.202467 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.201411 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7"] Oct 30 22:17:37.205524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.205494 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-db492e-6b5dbd8875-9kbz7"] Oct 30 22:17:37.217169 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.217140 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95"] Oct 30 22:17:37.614352 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.614292 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:37.790333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.785891 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:37.953129 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.953045 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b86dbf-0e5b-4c51-bb04-0a6292f15659" path="/var/lib/kubelet/pods/81b86dbf-0e5b-4c51-bb04-0a6292f15659/volumes" Oct 30 22:17:37.953531 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:37.953512 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac5c48b-8768-4e01-81c2-684c76dab0da" path="/var/lib/kubelet/pods/bac5c48b-8768-4e01-81c2-684c76dab0da/volumes" Oct 30 22:17:39.855395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:39.855357 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:40.375272 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:40.375237 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:17:41.870859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:41.870816 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:42.041615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.041582 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:42.217930 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.217839 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" podUID="6bb0ae00-2005-4a1a-b2d5-738e3718afa1" containerName="payload-processing" containerID="cri-o://6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08" gracePeriod=25 Oct 30 22:17:42.287379 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.287343 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:42.541102 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.541076 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:17:42.614909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.614882 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85\" (UniqueName: \"kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85\") pod \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " Oct 30 22:17:42.615034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.615017 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume\") pod \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\" (UID: \"6bb0ae00-2005-4a1a-b2d5-738e3718afa1\") " Oct 30 22:17:42.615399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.615376 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume" pod "6bb0ae00-2005-4a1a-b2d5-738e3718afa1" (UID: "6bb0ae00-2005-4a1a-b2d5-738e3718afa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:42.617087 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.617064 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85" pod "6bb0ae00-2005-4a1a-b2d5-738e3718afa1" (UID: "6bb0ae00-2005-4a1a-b2d5-738e3718afa1"). InnerVolumeSpecName "kube-api-access-zbm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:42.716995 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.716965 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:42.717122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.716998 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbm85\" (UniqueName: \"kubernetes.io/projected/6bb0ae00-2005-4a1a-b2d5-738e3718afa1-kube-api-access-zbm85\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:42.945776 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.945740 5508 generic.go:356] "Generic (PLEG): container finished" podID="6bb0ae00-2005-4a1a-b2d5-738e3718afa1" containerID="6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08" exitCode=0 Oct 30 22:17:42.945776 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.945775 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" Oct 30 22:17:42.946276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.945781 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" event={"ID":"6bb0ae00-2005-4a1a-b2d5-738e3718afa1","Type":"ContainerDied","Data":"6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08"} Oct 30 22:17:42.946276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.945826 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95" event={"ID":"6bb0ae00-2005-4a1a-b2d5-738e3718afa1","Type":"ContainerDied","Data":"e81f0f87b0a5d3908a7da86adc36348005efcacb55879987ee64802e4a789911"} Oct 30 22:17:42.946276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.945854 5508 scope.go:122] "RemoveContainer" containerID="6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08" Oct 30 22:17:42.956205 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.956174 5508 scope.go:122] "RemoveContainer" containerID="6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08" Oct 30 22:17:42.956524 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:42.956497 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08\": container with ID starting with 6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08 not found: ID does not exist" containerID="6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08" Oct 30 22:17:42.956598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:42.956534 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08"} err="failed to get container status \"6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08\": rpc error: code = NotFound desc = could not find container \"6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08\": container with ID starting with 6209010aac75deede5277d0a8bda3ebc557dbf815776dd045884243e3a2bfe08 not found: ID does not exist" Oct 30 22:17:43.109318 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:43.109274 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95"] Oct 30 22:17:43.112418 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:43.112392 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-5dff71-6d4fbd8b6c-57n95"] Oct 30 22:17:43.213932 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:43.213866 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.388160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:43.388127 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.953755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:43.953720 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb0ae00-2005-4a1a-b2d5-738e3718afa1" path="/var/lib/kubelet/pods/6bb0ae00-2005-4a1a-b2d5-738e3718afa1/volumes" Oct 30 22:17:45.016162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:45.016131 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.292014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:45.291926 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.463694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:45.463661 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.883516 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:45.883469 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:46.065157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:46.065120 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:50.372213 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:50.372122 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:54.213895 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.213862 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:54.458802 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.458770 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw"] Oct 30 22:17:54.458978 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.458960 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" podUID="2ddf0438-70cf-448d-9510-4fe6380d9937" containerName="maas-api" containerID="cri-o://a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc" gracePeriod=30 Oct 30 22:17:54.537794 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.537711 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:54.721613 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.721003 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:54.756622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.756599 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 22:17:54.869777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.868197 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\") pod \"2ddf0438-70cf-448d-9510-4fe6380d9937\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " Oct 30 22:17:54.869777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.868285 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs\") pod \"2ddf0438-70cf-448d-9510-4fe6380d9937\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " Oct 30 22:17:54.869777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.868413 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca\") pod \"2ddf0438-70cf-448d-9510-4fe6380d9937\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " Oct 30 22:17:54.894574 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.893899 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx\" (UniqueName: \"kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx\") pod \"2ddf0438-70cf-448d-9510-4fe6380d9937\" (UID: \"2ddf0438-70cf-448d-9510-4fe6380d9937\") " Oct 30 22:17:54.895217 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.895163 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca" pod "2ddf0438-70cf-448d-9510-4fe6380d9937" (UID: "2ddf0438-70cf-448d-9510-4fe6380d9937"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:54.934148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.919091 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx" pod "2ddf0438-70cf-448d-9510-4fe6380d9937" (UID: "2ddf0438-70cf-448d-9510-4fe6380d9937"). InnerVolumeSpecName "kube-api-access-x2znx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:54.934148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.919181 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs" pod "2ddf0438-70cf-448d-9510-4fe6380d9937" (UID: "2ddf0438-70cf-448d-9510-4fe6380d9937"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:54.934148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.919809 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls" pod "2ddf0438-70cf-448d-9510-4fe6380d9937" (UID: "2ddf0438-70cf-448d-9510-4fe6380d9937"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:54.996285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.996206 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddf0438-70cf-448d-9510-4fe6380d9937-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:54.996285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.996237 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2znx\" (UniqueName: \"kubernetes.io/projected/2ddf0438-70cf-448d-9510-4fe6380d9937-kube-api-access-x2znx\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:54.996285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.996255 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:54.996285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:54.996280 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ddf0438-70cf-448d-9510-4fe6380d9937-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:55.044901 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.044865 5508 generic.go:356] "Generic (PLEG): container finished" podID="2ddf0438-70cf-448d-9510-4fe6380d9937" containerID="a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc" exitCode=0 Oct 30 22:17:55.045181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.044906 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" event={"ID":"2ddf0438-70cf-448d-9510-4fe6380d9937","Type":"ContainerDied","Data":"a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc"} Oct 30 22:17:55.045332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.044924 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" Oct 30 22:17:55.045332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.045222 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw" event={"ID":"2ddf0438-70cf-448d-9510-4fe6380d9937","Type":"ContainerDied","Data":"1b1e4f5dc69f7c3de6ca576b72a487dcd2992261cc3179c76217d968f600fca9"} Oct 30 22:17:55.045332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.045257 5508 scope.go:122] "RemoveContainer" containerID="a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc" Oct 30 22:17:55.061680 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.061657 5508 scope.go:122] "RemoveContainer" containerID="a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc" Oct 30 22:17:55.061998 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:17:55.061967 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc\": container with ID starting with a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc not found: ID does not exist" containerID="a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc" Oct 30 22:17:55.062092 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.061999 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc"} err="failed to get container status \"a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc\": rpc error: code = NotFound desc = could not find container \"a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc\": container with ID starting with a370e9b441eb0f1ec3780d78939b252edba92ce701d59598e29a3661c15722dc not found: ID does not exist" Oct 30 22:17:55.208991 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.208957 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw"] Oct 30 22:17:55.211491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.211464 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-0c99b1-7fbccc8957-7gqsw"] Oct 30 22:17:55.796757 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.796710 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:55.952959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:55.952925 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddf0438-70cf-448d-9510-4fe6380d9937" path="/var/lib/kubelet/pods/2ddf0438-70cf-448d-9510-4fe6380d9937/volumes" Oct 30 22:17:56.169260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:56.169219 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:56.463031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:56.462954 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:57.065545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:57.065508 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:57.434719 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:57.434684 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:57.683190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:57.683148 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:57.849271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:57.849137 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:58.332649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:17:58.332615 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:02.047761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:02.047721 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:03.574216 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:03.574149 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:03.891012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:03.890907 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:03.976998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:03.976950 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk"] Oct 30 22:18:03.977190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:03.977170 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" containerID="cri-o://647368fdf6e1f8b25ffa360ad743b660ffa8c1fea82afbbd2bf2549f9638b6b1" gracePeriod=30 Oct 30 22:18:04.079340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:04.075656 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:04.629533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:04.629497 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:05.375319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:05.375282 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:05.548346 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:05.548288 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:05.695592 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:05.695511 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9"] Oct 30 22:18:05.696008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:05.695767 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" containerID="cri-o://10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079" gracePeriod=30 Oct 30 22:18:05.736980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:05.736954 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:06.173741 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:06.173707 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:09.126737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.126707 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:09.194649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.193147 5508 generic.go:356] "Generic (PLEG): container finished" podID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerID="647368fdf6e1f8b25ffa360ad743b660ffa8c1fea82afbbd2bf2549f9638b6b1" exitCode=0 Oct 30 22:18:09.195530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.195501 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" event={"ID":"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d","Type":"ContainerDied","Data":"647368fdf6e1f8b25ffa360ad743b660ffa8c1fea82afbbd2bf2549f9638b6b1"} Oct 30 22:18:09.276860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.276837 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:18:09.344143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344115 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344182 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344205 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344235 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344257 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344394 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344417 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.344540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344467 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344506 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344555 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.344756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344636 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket\") pod \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\" (UID: \"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d\") " Oct 30 22:18:09.344756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344657 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.344756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344681 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:09.344965 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.344939 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.345394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.345369 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.345539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.345398 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.345539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.345416 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.345539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.345432 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.345539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.345447 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.346640 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.346617 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.346750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.346727 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "kube-api-access-d9kc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:09.346796 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.346764 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:09.346834 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.346819 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo" pod "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" (UID: "fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:09.447178 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.447144 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.447178 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.447179 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kc2\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-kube-api-access-d9kc2\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.448195 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.447193 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.448195 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:09.447205 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.203114 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.203075 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" event={"ID":"fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d","Type":"ContainerDied","Data":"5e74161a4c942e26975cfc848542a0d26c7c3bd12ea0b7aeb01195e1915553b2"} Oct 30 22:18:10.203576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.203130 5508 scope.go:122] "RemoveContainer" containerID="647368fdf6e1f8b25ffa360ad743b660ffa8c1fea82afbbd2bf2549f9638b6b1" Oct 30 22:18:10.203576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.203083 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk" Oct 30 22:18:10.374087 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:10.374050 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:18:10.416160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.416119 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk"] Oct 30 22:18:10.465000 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.464895 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-mn4hk"] Oct 30 22:18:10.546942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.546913 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:10.812980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.812951 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:10.815157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.815134 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs"] Oct 30 22:18:10.815396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.815369 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" containerID="cri-o://5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4" gracePeriod=30 Oct 30 22:18:10.818395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.818368 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9"] Oct 30 22:18:10.818731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.818573 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" containerID="cri-o://dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6" gracePeriod=30 Oct 30 22:18:10.981589 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:10.981450 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:11.032940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.032916 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:18:11.076420 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076388 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076449 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076517 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076571 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076593 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076617 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076653 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076651 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:11.076823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076680 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076786 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo\") pod \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\" (UID: \"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9\") " Oct 30 22:18:11.076823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076793 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:11.077199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076915 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:11.077199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.076970 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:11.077199 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.077027 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:11.077389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.077338 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.077389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.077357 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.077389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.077371 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.077389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.077384 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.077781 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.077397 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.079134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.079109 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:11.079134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.079124 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:11.079535 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.079514 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "kube-api-access-hx6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:11.079650 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.079535 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token" pod "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" (UID: "676bfdef-67cf-43fb-8c33-2b6a06f6c4f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:11.133878 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.133848 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m"] Oct 30 22:18:11.178844 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.178810 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.178844 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.178838 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6lv\" (UniqueName: \"kubernetes.io/projected/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-kube-api-access-hx6lv\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.179023 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.178857 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.179023 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.178873 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:11.223005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.222960 5508 generic.go:356] "Generic (PLEG): container finished" podID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerID="10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079" exitCode=0 Oct 30 22:18:11.223539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.223053 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" event={"ID":"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9","Type":"ContainerDied","Data":"10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079"} Oct 30 22:18:11.223539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.223139 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" event={"ID":"676bfdef-67cf-43fb-8c33-2b6a06f6c4f9","Type":"ContainerDied","Data":"bbbe8df00f3dbf09f7743cfe086e397385fed812f752bcbe8380256d3e61ba25"} Oct 30 22:18:11.223539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.223359 5508 scope.go:122] "RemoveContainer" containerID="10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079" Oct 30 22:18:11.223539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.223387 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9" Oct 30 22:18:11.236065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.236044 5508 scope.go:122] "RemoveContainer" containerID="dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976" Oct 30 22:18:11.245457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.245431 5508 scope.go:122] "RemoveContainer" containerID="10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079" Oct 30 22:18:11.245734 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:11.245707 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079\": container with ID starting with 10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079 not found: ID does not exist" containerID="10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079" Oct 30 22:18:11.245829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.245747 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079"} err="failed to get container status \"10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079\": rpc error: code = NotFound desc = could not find container \"10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079\": container with ID starting with 10c6aec6f7bc1854b709fae4f8b278d742cfee4a06ba6190ccd1ed3f4ab6b079 not found: ID does not exist" Oct 30 22:18:11.245829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.245767 5508 scope.go:122] "RemoveContainer" containerID="dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976" Oct 30 22:18:11.246078 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:11.246052 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976\": container with ID starting with dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976 not found: ID does not exist" containerID="dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976" Oct 30 22:18:11.246126 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.246090 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976"} err="failed to get container status \"dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976\": rpc error: code = NotFound desc = could not find container \"dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976\": container with ID starting with dd826ea292659aab4d12bceff60c37df9b4d7c7331fdcb3ad8821d6497396976 not found: ID does not exist" Oct 30 22:18:11.270470 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.270432 5508 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:11.270631 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.270501 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:11.314851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.314813 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:11.421986 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.421953 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9"] Oct 30 22:18:11.426210 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.426184 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-db492e-openshift-default-76c67c5cbf-z7bp9"] Oct 30 22:18:11.480972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.480939 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:11.953340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.953274 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" path="/var/lib/kubelet/pods/676bfdef-67cf-43fb-8c33-2b6a06f6c4f9/volumes" Oct 30 22:18:11.953869 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:11.953851 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" path="/var/lib/kubelet/pods/fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d/volumes" Oct 30 22:18:14.527548 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:14.527496 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:15.928821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:15.928792 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:16.101339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.101288 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:18:16.118497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.118477 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:18:16.238327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238267 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.238567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238343 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238374 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238461 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.238567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238490 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238567 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238560 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238588 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238644 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238686 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238727 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238753 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238770 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238798 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238817 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.238864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238856 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.239209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238875 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.239209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238895 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert\") pod \"54eb8069-49be-41e8-a94c-26d8e51710aa\" (UID: \"54eb8069-49be-41e8-a94c-26d8e51710aa\") " Oct 30 22:18:16.239209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238912 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.239209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.238936 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket\") pod \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\" (UID: \"8f9c240d-6eca-4f39-8f16-0ce7a40cf274\") " Oct 30 22:18:16.239897 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.239529 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.239981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.239710 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.240170 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.239841 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.240712 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.240568 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.241032 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.241006 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:16.242525 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.242498 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.242835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.242813 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:16.244377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.243060 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.244377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.243244 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.244377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.243440 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.244780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.244749 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.244880 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.244822 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:16.244880 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.244862 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.244993 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.244961 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:16.245072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.245052 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.245828 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.245806 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt"] Oct 30 22:18:16.246026 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.245999 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "kube-api-access-qj2dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.246117 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.246030 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b" pod "8f9c240d-6eca-4f39-8f16-0ce7a40cf274" (UID: "8f9c240d-6eca-4f39-8f16-0ce7a40cf274"). InnerVolumeSpecName "kube-api-access-r7b6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.246774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.246745 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token" pod "54eb8069-49be-41e8-a94c-26d8e51710aa" (UID: "54eb8069-49be-41e8-a94c-26d8e51710aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.255222 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.255191 5508 generic.go:356] "Generic (PLEG): container finished" podID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerID="5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4" exitCode=0 Oct 30 22:18:16.255356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.255297 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" Oct 30 22:18:16.255411 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.255369 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" event={"ID":"8f9c240d-6eca-4f39-8f16-0ce7a40cf274","Type":"ContainerDied","Data":"5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4"} Oct 30 22:18:16.255464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.255429 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs" event={"ID":"8f9c240d-6eca-4f39-8f16-0ce7a40cf274","Type":"ContainerDied","Data":"417f890df9f077f64b4c0ea55bac98260c6281c221b6361a2061267c4f6bc570"} Oct 30 22:18:16.255464 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.255461 5508 scope.go:122] "RemoveContainer" containerID="5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4" Oct 30 22:18:16.256995 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.256966 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" Oct 30 22:18:16.257090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.257022 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" event={"ID":"54eb8069-49be-41e8-a94c-26d8e51710aa","Type":"ContainerDied","Data":"dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6"} Oct 30 22:18:16.257090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.256977 5508 generic.go:356] "Generic (PLEG): container finished" podID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerID="dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6" exitCode=0 Oct 30 22:18:16.257090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.257064 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9" event={"ID":"54eb8069-49be-41e8-a94c-26d8e51710aa","Type":"ContainerDied","Data":"a67d894ed5ce249ad9718a34dccb9884e7957a14eedba127474a33806fe8a492"} Oct 30 22:18:16.267349 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.267189 5508 scope.go:122] "RemoveContainer" containerID="4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343" Oct 30 22:18:16.276959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.276935 5508 scope.go:122] "RemoveContainer" containerID="5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4" Oct 30 22:18:16.277286 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:16.277261 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4\": container with ID starting with 5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4 not found: ID does not exist" containerID="5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4" Oct 30 22:18:16.277426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.277293 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4"} err="failed to get container status \"5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4\": rpc error: code = NotFound desc = could not find container \"5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4\": container with ID starting with 5377e069d617721136b0a5cb7bf8726f5a5bdb059f1dd30136e272b3c65cbce4 not found: ID does not exist" Oct 30 22:18:16.277426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.277345 5508 scope.go:122] "RemoveContainer" containerID="4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343" Oct 30 22:18:16.277773 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:16.277753 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343\": container with ID starting with 4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343 not found: ID does not exist" containerID="4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343" Oct 30 22:18:16.277827 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.277782 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343"} err="failed to get container status \"4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343\": rpc error: code = NotFound desc = could not find container \"4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343\": container with ID starting with 4dc64b90beca06388232d4f013d662907e8f267ff5068494bf88caa83357d343 not found: ID does not exist" Oct 30 22:18:16.277827 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.277806 5508 scope.go:122] "RemoveContainer" containerID="dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6" Oct 30 22:18:16.286504 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.286487 5508 scope.go:122] "RemoveContainer" containerID="dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6" Oct 30 22:18:16.286767 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:16.286747 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6\": container with ID starting with dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6 not found: ID does not exist" containerID="dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6" Oct 30 22:18:16.286818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.286773 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6"} err="failed to get container status \"dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6\": rpc error: code = NotFound desc = could not find container \"dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6\": container with ID starting with dd9e01d4b0f72789b0c68bfffda87ba742babeba503fde1c39fcabd3345033e6 not found: ID does not exist" Oct 30 22:18:16.340854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340789 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54eb8069-49be-41e8-a94c-26d8e51710aa-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.340854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340814 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.340854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340825 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.340854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340834 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.340854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340843 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj2dz\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-kube-api-access-qj2dz\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.340854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340852 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340860 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340869 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340878 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340886 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340894 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340901 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340909 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340917 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340925 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7b6b\" (UniqueName: \"kubernetes.io/projected/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-kube-api-access-r7b6b\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340933 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f9c240d-6eca-4f39-8f16-0ce7a40cf274-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.341131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.340941 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54eb8069-49be-41e8-a94c-26d8e51710aa-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.357154 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.357128 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:16.495749 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.495714 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs"] Oct 30 22:18:16.500079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.500042 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-pltcs"] Oct 30 22:18:16.678389 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.678276 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9"] Oct 30 22:18:16.683785 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:16.683758 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-r87z9"] Oct 30 22:18:17.952016 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:17.951983 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" path="/var/lib/kubelet/pods/54eb8069-49be-41e8-a94c-26d8e51710aa/volumes" Oct 30 22:18:17.952499 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:17.952482 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" path="/var/lib/kubelet/pods/8f9c240d-6eca-4f39-8f16-0ce7a40cf274/volumes" Oct 30 22:18:18.668271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:18.668233 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:18.826630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:18.826597 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:19.664562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:19.664496 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:22.314058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:22.314018 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:23.733972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:23.733933 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:23.960347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:23.960300 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:24.766656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:24.766620 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:26.154722 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.154667 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" containerID="cri-o://0c3d9cf8bb4b0c2315ac07a151c6695be0765c671d53afafcd47138b76a136c9" gracePeriod=45 Oct 30 22:18:26.322928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.322895 5508 generic.go:356] "Generic (PLEG): container finished" podID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerID="0c3d9cf8bb4b0c2315ac07a151c6695be0765c671d53afafcd47138b76a136c9" exitCode=0 Oct 30 22:18:26.323096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.322961 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" event={"ID":"5dd33b67-01de-45df-a7af-2920af1ca4ec","Type":"ContainerDied","Data":"0c3d9cf8bb4b0c2315ac07a151c6695be0765c671d53afafcd47138b76a136c9"} Oct 30 22:18:26.420511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.420456 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:18:26.459554 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459523 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir\") pod \"5dd33b67-01de-45df-a7af-2920af1ca4ec\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " Oct 30 22:18:26.459677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459562 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home\") pod \"5dd33b67-01de-45df-a7af-2920af1ca4ec\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " Oct 30 22:18:26.459677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459632 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs\") pod \"5dd33b67-01de-45df-a7af-2920af1ca4ec\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " Oct 30 22:18:26.459677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459656 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm\") pod \"5dd33b67-01de-45df-a7af-2920af1ca4ec\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " Oct 30 22:18:26.459792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459704 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache\") pod \"5dd33b67-01de-45df-a7af-2920af1ca4ec\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " Oct 30 22:18:26.459868 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459841 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir" pod "5dd33b67-01de-45df-a7af-2920af1ca4ec" (UID: "5dd33b67-01de-45df-a7af-2920af1ca4ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.459909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459876 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home" pod "5dd33b67-01de-45df-a7af-2920af1ca4ec" (UID: "5dd33b67-01de-45df-a7af-2920af1ca4ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.459940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459901 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695\" (UniqueName: \"kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695\") pod \"5dd33b67-01de-45df-a7af-2920af1ca4ec\" (UID: \"5dd33b67-01de-45df-a7af-2920af1ca4ec\") " Oct 30 22:18:26.459989 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.459970 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache" pod "5dd33b67-01de-45df-a7af-2920af1ca4ec" (UID: "5dd33b67-01de-45df-a7af-2920af1ca4ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.460465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.460352 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.460465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.460371 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.460465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.460380 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.462069 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.462037 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm" pod "5dd33b67-01de-45df-a7af-2920af1ca4ec" (UID: "5dd33b67-01de-45df-a7af-2920af1ca4ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.462069 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.462045 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695" pod "5dd33b67-01de-45df-a7af-2920af1ca4ec" (UID: "5dd33b67-01de-45df-a7af-2920af1ca4ec"). InnerVolumeSpecName "kube-api-access-69695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.462201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.462119 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs" pod "5dd33b67-01de-45df-a7af-2920af1ca4ec" (UID: "5dd33b67-01de-45df-a7af-2920af1ca4ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:26.562300 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.562262 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd33b67-01de-45df-a7af-2920af1ca4ec-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.562300 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.562328 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd33b67-01de-45df-a7af-2920af1ca4ec-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.562526 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:26.562346 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69695\" (UniqueName: \"kubernetes.io/projected/5dd33b67-01de-45df-a7af-2920af1ca4ec-kube-api-access-69695\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:27.329226 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:27.329197 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" Oct 30 22:18:27.329727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:27.329225 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m" event={"ID":"5dd33b67-01de-45df-a7af-2920af1ca4ec","Type":"ContainerDied","Data":"02d5c1b7fde236126b31bfa5fa5b98be0f7392ffd1d87860bdda7456bfec0b48"} Oct 30 22:18:27.329727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:27.329275 5508 scope.go:122] "RemoveContainer" containerID="0c3d9cf8bb4b0c2315ac07a151c6695be0765c671d53afafcd47138b76a136c9" Oct 30 22:18:27.493556 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:27.493520 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m"] Oct 30 22:18:27.542505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:27.542476 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/premium-w0-dde1e8-kserve-546445bc9-2dq6m"] Oct 30 22:18:27.951854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:27.951821 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" path="/var/lib/kubelet/pods/5dd33b67-01de-45df-a7af-2920af1ca4ec/volumes" Oct 30 22:18:29.920806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:29.920772 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:31.270436 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.270395 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" containerID="cri-o://88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767" gracePeriod=45 Oct 30 22:18:31.527396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.527334 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:18:31.618805 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.618779 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir\") pod \"34d1011f-485e-4152-af59-fc2f37d02649\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " Oct 30 22:18:31.618939 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.618912 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm\") pod \"34d1011f-485e-4152-af59-fc2f37d02649\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " Oct 30 22:18:31.618980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.618957 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688\" (UniqueName: \"kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688\") pod \"34d1011f-485e-4152-af59-fc2f37d02649\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " Oct 30 22:18:31.619015 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.618986 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache\") pod \"34d1011f-485e-4152-af59-fc2f37d02649\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " Oct 30 22:18:31.619046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619018 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir" pod "34d1011f-485e-4152-af59-fc2f37d02649" (UID: "34d1011f-485e-4152-af59-fc2f37d02649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.619085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619065 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs\" (UniqueName: \"kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs\") pod \"34d1011f-485e-4152-af59-fc2f37d02649\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " Oct 30 22:18:31.619115 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619096 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home\") pod \"34d1011f-485e-4152-af59-fc2f37d02649\" (UID: \"34d1011f-485e-4152-af59-fc2f37d02649\") " Oct 30 22:18:31.619247 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619221 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache" pod "34d1011f-485e-4152-af59-fc2f37d02649" (UID: "34d1011f-485e-4152-af59-fc2f37d02649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.619455 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619432 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home" pod "34d1011f-485e-4152-af59-fc2f37d02649" (UID: "34d1011f-485e-4152-af59-fc2f37d02649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.619795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619694 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.619860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619810 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.619860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.619827 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.621260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.621237 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm" pod "34d1011f-485e-4152-af59-fc2f37d02649" (UID: "34d1011f-485e-4152-af59-fc2f37d02649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.621359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.621254 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs" pod "34d1011f-485e-4152-af59-fc2f37d02649" (UID: "34d1011f-485e-4152-af59-fc2f37d02649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:31.621359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.621345 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688" pod "34d1011f-485e-4152-af59-fc2f37d02649" (UID: "34d1011f-485e-4152-af59-fc2f37d02649"). InnerVolumeSpecName "kube-api-access-kz688". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:31.721033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.721002 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34d1011f-485e-4152-af59-fc2f37d02649-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.721033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.721031 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34d1011f-485e-4152-af59-fc2f37d02649-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.721223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:31.721041 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz688\" (UniqueName: \"kubernetes.io/projected/34d1011f-485e-4152-af59-fc2f37d02649-kube-api-access-kz688\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:32.367101 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.367067 5508 generic.go:356] "Generic (PLEG): container finished" podID="34d1011f-485e-4152-af59-fc2f37d02649" containerID="88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767" exitCode=0 Oct 30 22:18:32.367649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.367110 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" Oct 30 22:18:32.367649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.367158 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" event={"ID":"34d1011f-485e-4152-af59-fc2f37d02649","Type":"ContainerDied","Data":"88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767"} Oct 30 22:18:32.367649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.367198 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt" event={"ID":"34d1011f-485e-4152-af59-fc2f37d02649","Type":"ContainerDied","Data":"c75285717c1370fa15dfff88bd5e97ea03b0628b845819efd2e94978675fc812"} Oct 30 22:18:32.367649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.367215 5508 scope.go:122] "RemoveContainer" containerID="88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767" Oct 30 22:18:32.376187 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.376166 5508 scope.go:122] "RemoveContainer" containerID="88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767" Oct 30 22:18:32.376465 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:32.376444 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767\": container with ID starting with 88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767 not found: ID does not exist" containerID="88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767" Oct 30 22:18:32.376524 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.376471 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767"} err="failed to get container status \"88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767\": rpc error: code = NotFound desc = could not find container \"88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767\": container with ID starting with 88c793a0fa5483a8078b1acf2e4c85395f8dce08b3082e5ae13300b1b0419767 not found: ID does not exist" Oct 30 22:18:32.530503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.530470 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt"] Oct 30 22:18:32.579858 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:32.579824 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/premium-w5-9f8aea-kserve-86647694d7-7jhlt"] Oct 30 22:18:33.201913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:33.201880 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:33.451754 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:33.451694 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:33.952017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:33.951986 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34d1011f-485e-4152-af59-fc2f37d02649" path="/var/lib/kubelet/pods/34d1011f-485e-4152-af59-fc2f37d02649/volumes" Oct 30 22:18:35.585748 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:35.585713 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:40.372677 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:40.372636 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:18:40.672722 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:40.672639 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:41.213166 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:41.213131 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:41.413622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:41.413585 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:45.001689 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:45.001649 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:45.171972 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:45.171929 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:45.597667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:45.597634 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:48.263289 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:48.263258 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:48.429773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:48.429738 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:50.584724 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:50.584686 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:50.756054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:50.756022 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:50.924251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:50.924174 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:52.642410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:52.642375 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:52.827177 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:52.827131 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:53.004534 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:53.004434 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:54.480767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:54.480731 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:55.043811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:55.043768 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:55.943721 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:55.943675 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v"] Oct 30 22:18:55.944179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:55.943947 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" podUID="03672869-8c66-4958-8597-8be4a451981b" containerName="maas-api" containerID="cri-o://366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813" gracePeriod=30 Oct 30 22:18:56.069534 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.069503 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:56.240673 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.240649 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 22:18:56.354871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.353784 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca\") pod \"03672869-8c66-4958-8597-8be4a451981b\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " Oct 30 22:18:56.354871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.353867 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v\" (UniqueName: \"kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v\") pod \"03672869-8c66-4958-8597-8be4a451981b\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " Oct 30 22:18:56.354871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.353945 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs\") pod \"03672869-8c66-4958-8597-8be4a451981b\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " Oct 30 22:18:56.354871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.353986 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\") pod \"03672869-8c66-4958-8597-8be4a451981b\" (UID: \"03672869-8c66-4958-8597-8be4a451981b\") " Oct 30 22:18:56.356137 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.356092 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca" pod "03672869-8c66-4958-8597-8be4a451981b" (UID: "03672869-8c66-4958-8597-8be4a451981b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:56.359764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.359686 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs" pod "03672869-8c66-4958-8597-8be4a451981b" (UID: "03672869-8c66-4958-8597-8be4a451981b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:56.359764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.359742 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls" pod "03672869-8c66-4958-8597-8be4a451981b" (UID: "03672869-8c66-4958-8597-8be4a451981b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:56.359940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.359776 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v" pod "03672869-8c66-4958-8597-8be4a451981b" (UID: "03672869-8c66-4958-8597-8be4a451981b"). InnerVolumeSpecName "kube-api-access-r987v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:56.455912 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.455880 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:56.455912 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.455907 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03672869-8c66-4958-8597-8be4a451981b-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:56.456085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.455919 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03672869-8c66-4958-8597-8be4a451981b-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:56.456085 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.455928 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r987v\" (UniqueName: \"kubernetes.io/projected/03672869-8c66-4958-8597-8be4a451981b-kube-api-access-r987v\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:56.458185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.458161 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:56.500861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.500776 5508 generic.go:356] "Generic (PLEG): container finished" podID="03672869-8c66-4958-8597-8be4a451981b" containerID="366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813" exitCode=0 Oct 30 22:18:56.500861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.500831 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" Oct 30 22:18:56.500861 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.500844 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" event={"ID":"03672869-8c66-4958-8597-8be4a451981b","Type":"ContainerDied","Data":"366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813"} Oct 30 22:18:56.501093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.500891 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v" event={"ID":"03672869-8c66-4958-8597-8be4a451981b","Type":"ContainerDied","Data":"e579698b6a9f84ba26a2923be4663ce16df63c3b4b7cb7adb9bc6c8d35672b74"} Oct 30 22:18:56.501093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.500917 5508 scope.go:122] "RemoveContainer" containerID="366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813" Oct 30 22:18:56.511187 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.511168 5508 scope.go:122] "RemoveContainer" containerID="366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813" Oct 30 22:18:56.511505 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:18:56.511483 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813\": container with ID starting with 366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813 not found: ID does not exist" containerID="366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813" Oct 30 22:18:56.511565 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.511515 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813"} err="failed to get container status \"366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813\": rpc error: code = NotFound desc = could not find container \"366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813\": container with ID starting with 366513680ca1f3f5ed117adb37e64e40524d04c959ca74a8dfee4accb1527813 not found: ID does not exist" Oct 30 22:18:56.662885 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.662856 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v"] Oct 30 22:18:56.666198 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.666173 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9f8aea-656659d7d-ggc8v"] Oct 30 22:18:56.737429 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:56.737402 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:57.371250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:57.371222 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:57.544289 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:57.544255 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:57.706942 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:57.706847 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:57.952666 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:57.952623 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03672869-8c66-4958-8597-8be4a451981b" path="/var/lib/kubelet/pods/03672869-8c66-4958-8597-8be4a451981b/volumes" Oct 30 22:18:58.036093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:58.036022 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:58.226599 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:58.226562 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:58.723833 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:18:58.723795 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:00.355821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:00.355781 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:02.859008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:02.858966 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:03.748119 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:03.748084 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:03.926463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:03.926431 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.011690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:04.011617 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9"] Oct 30 22:19:04.011841 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:04.011821 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" containerID="cri-o://2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2" gracePeriod=30 Oct 30 22:19:04.105999 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:04.105949 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.409790 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:04.409759 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.682697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:04.682626 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.884728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:04.884688 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:05.084895 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:05.084863 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:05.321472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:05.321437 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst"] Oct 30 22:19:05.321683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:05.321658 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" containerID="cri-o://1480aabcb1100a737e3edb8eb311fd459ef3684a77ea5425db63d036624daaad" gracePeriod=30 Oct 30 22:19:05.334749 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:05.334722 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:05.504231 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:05.504150 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:06.523492 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:06.523455 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:06.772613 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:06.772576 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:09.278610 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.278550 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 22:19:09.327794 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.327767 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.327929 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.327850 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.327929 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.327899 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.327932 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.327962 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328001 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328050 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:09.328157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328091 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:09.328157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328097 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328173 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328271 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token\") pod \"86590685-ff39-41c3-a263-b54d75714605\" (UID: \"86590685-ff39-41c3-a263-b54d75714605\") " Oct 30 22:19:09.328342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328281 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:09.328543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328348 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:09.328617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328593 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:09.328816 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328793 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.328945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328820 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.328945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328833 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.328945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328842 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86590685-ff39-41c3-a263-b54d75714605-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.328945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.328850 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.330731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.330709 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:09.330731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.330724 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:09.330926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.330908 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "kube-api-access-mcz22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:09.330977 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.330920 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token" pod "86590685-ff39-41c3-a263-b54d75714605" (UID: "86590685-ff39-41c3-a263-b54d75714605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:09.430181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.430139 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.430181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.430166 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86590685-ff39-41c3-a263-b54d75714605-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.430181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.430176 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86590685-ff39-41c3-a263-b54d75714605-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.430181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.430187 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcz22\" (UniqueName: \"kubernetes.io/projected/86590685-ff39-41c3-a263-b54d75714605-kube-api-access-mcz22\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:09.574627 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.574588 5508 generic.go:356] "Generic (PLEG): container finished" podID="86590685-ff39-41c3-a263-b54d75714605" containerID="2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2" exitCode=0 Oct 30 22:19:09.574821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.574662 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" Oct 30 22:19:09.574821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.574672 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" event={"ID":"86590685-ff39-41c3-a263-b54d75714605","Type":"ContainerDied","Data":"2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2"} Oct 30 22:19:09.574821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.574774 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9" event={"ID":"86590685-ff39-41c3-a263-b54d75714605","Type":"ContainerDied","Data":"54dd9cd9ee7b287ef3fcf12f4773dab04c69b69e9df9ea807d309ba5274da9f2"} Oct 30 22:19:09.574821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.574805 5508 scope.go:122] "RemoveContainer" containerID="2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2" Oct 30 22:19:09.584832 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.584813 5508 scope.go:122] "RemoveContainer" containerID="112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01" Oct 30 22:19:09.593504 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.593488 5508 scope.go:122] "RemoveContainer" containerID="2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2" Oct 30 22:19:09.593786 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:19:09.593766 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2\": container with ID starting with 2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2 not found: ID does not exist" containerID="2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2" Oct 30 22:19:09.593849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.593797 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2"} err="failed to get container status \"2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2\": rpc error: code = NotFound desc = could not find container \"2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2\": container with ID starting with 2c59da009da3968932763ccae88d3d4f63ebd5a65482480e688dc944159283b2 not found: ID does not exist" Oct 30 22:19:09.593849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.593815 5508 scope.go:122] "RemoveContainer" containerID="112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01" Oct 30 22:19:09.594075 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:19:09.594057 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01\": container with ID starting with 112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01 not found: ID does not exist" containerID="112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01" Oct 30 22:19:09.594125 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.594082 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01"} err="failed to get container status \"112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01\": rpc error: code = NotFound desc = could not find container \"112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01\": container with ID starting with 112340731e188e6699c2c6dc33d0c010fba7bc8060dc48621aa6a976d286ed01 not found: ID does not exist" Oct 30 22:19:09.739397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.739350 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9"] Oct 30 22:19:09.740983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.740963 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2djh9"] Oct 30 22:19:09.887453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.887372 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:09.952392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:09.952360 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86590685-ff39-41c3-a263-b54d75714605" path="/var/lib/kubelet/pods/86590685-ff39-41c3-a263-b54d75714605/volumes" Oct 30 22:19:10.358587 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.358548 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:10.375870 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:19:10.375834 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:19:10.583684 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.583657 5508 generic.go:356] "Generic (PLEG): container finished" podID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerID="1480aabcb1100a737e3edb8eb311fd459ef3684a77ea5425db63d036624daaad" exitCode=0 Oct 30 22:19:10.583881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.583690 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" event={"ID":"e5bf880a-64d5-481b-8cc5-bde91bb82995","Type":"ContainerDied","Data":"1480aabcb1100a737e3edb8eb311fd459ef3684a77ea5425db63d036624daaad"} Oct 30 22:19:10.583881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.583733 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" event={"ID":"e5bf880a-64d5-481b-8cc5-bde91bb82995","Type":"ContainerDied","Data":"65b858ebaec9140370741f6b44a0464560c53c20c40249228032e6d31ab96923"} Oct 30 22:19:10.583881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.583751 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b858ebaec9140370741f6b44a0464560c53c20c40249228032e6d31ab96923" Oct 30 22:19:10.583989 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.583912 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 22:19:10.642503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642428 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642616 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642559 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642616 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642587 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642625 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642674 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642676 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.642727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642695 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642714 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642740 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642783 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token\") pod \"e5bf880a-64d5-481b-8cc5-bde91bb82995\" (UID: \"e5bf880a-64d5-481b-8cc5-bde91bb82995\") " Oct 30 22:19:10.642984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642821 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.642984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.642898 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.643171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643006 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.643171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643096 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:10.643301 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643284 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.643391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643344 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf880a-64d5-481b-8cc5-bde91bb82995-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.643391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643361 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.643391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643375 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.643391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.643388 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.645065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.645043 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.645161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.645072 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:10.645502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.645488 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "kube-api-access-md5m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:10.645552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.645509 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo" pod "e5bf880a-64d5-481b-8cc5-bde91bb82995" (UID: "e5bf880a-64d5-481b-8cc5-bde91bb82995"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:10.744767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.744734 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.744767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.744761 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.744767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.744770 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.744978 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.744784 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md5m5\" (UniqueName: \"kubernetes.io/projected/e5bf880a-64d5-481b-8cc5-bde91bb82995-kube-api-access-md5m5\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.823074 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:10.823046 5508 scope.go:122] "RemoveContainer" containerID="1480aabcb1100a737e3edb8eb311fd459ef3684a77ea5425db63d036624daaad" Oct 30 22:19:11.589358 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:11.589325 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst" Oct 30 22:19:11.868619 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:11.868528 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst"] Oct 30 22:19:11.873142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:11.873111 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-4nlst"] Oct 30 22:19:11.952106 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:11.952070 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" path="/var/lib/kubelet/pods/e5bf880a-64d5-481b-8cc5-bde91bb82995/volumes" Oct 30 22:19:14.674610 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:14.674565 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:19.967422 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:19.967393 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:24.424200 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:24.424165 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:24.587250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:24.587212 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:29.463837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:29.463805 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:30.065642 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:30.065607 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:30.240082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:30.240046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:33.601080 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:33.601041 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:34.259161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:34.259118 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:39.190590 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:39.190553 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:40.372569 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:19:40.372530 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:19:43.977751 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:43.977715 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:46.072598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:46.072564 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:47.494192 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:47.494156 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:47.786790 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:47.786703 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:49.203943 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:49.203906 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:51.006671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:51.006638 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc"] Oct 30 22:19:51.056207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:51.056177 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh"] Oct 30 22:19:51.176487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:51.176454 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:53.299271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:53.299232 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:54.365280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:54.365225 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:57.924340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:57.923920 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt"] Oct 30 22:19:57.924340 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:57.924158 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" containerID="cri-o://20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee" gracePeriod=30 Oct 30 22:19:57.978010 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:57.977968 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg"] Oct 30 22:19:57.978396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:57.978347 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" containerID="cri-o://c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd" gracePeriod=30 Oct 30 22:19:58.022590 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:58.022552 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt"] Oct 30 22:19:58.471850 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:58.471811 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:58.642367 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:58.642329 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:59.474125 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:59.474084 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:59.877135 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:19:59.877097 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:00.044028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:00.043993 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:00.229854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:00.229760 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:00.481164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:00.481086 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:20:00.481164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:00.481143 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:20:00.610036 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:00.610005 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:20:00.610202 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:00.610055 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:20:03.023397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.023355 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerName="payload-processing" containerID="cri-o://1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4" gracePeriod=25 Oct 30 22:20:03.249355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.249327 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:20:03.274690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.274660 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:20:03.312421 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312383 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312623 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312506 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312623 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312556 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312623 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312597 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312643 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312672 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312706 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312735 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.312793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312744 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.313042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.312991 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.313042 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313010 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.313136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313108 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:03.313172 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313141 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket\") pod \"6043a15c-2b8e-44ae-af90-9f75919bc43c\" (UID: \"6043a15c-2b8e-44ae-af90-9f75919bc43c\") " Oct 30 22:20:03.313465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313439 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.313756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313617 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6043a15c-2b8e-44ae-af90-9f75919bc43c-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.313756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313630 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.313756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313640 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.313756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313658 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.313756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.313672 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.315165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.315135 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.315268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.315244 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "kube-api-access-lbftb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:03.315542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.315521 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:03.315593 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.315572 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo" pod "6043a15c-2b8e-44ae-af90-9f75919bc43c" (UID: "6043a15c-2b8e-44ae-af90-9f75919bc43c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:03.414902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.414856 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.414908 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.414948 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.414992 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415035 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415074 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415123 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415180 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415366 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415264 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data\") pod \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\" (UID: \"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46\") " Oct 30 22:20:03.415871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415526 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.415871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415729 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.415871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415533 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415886 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415903 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415915 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415928 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415941 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbftb\" (UniqueName: \"kubernetes.io/projected/6043a15c-2b8e-44ae-af90-9f75919bc43c-kube-api-access-lbftb\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415953 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.415964 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6043a15c-2b8e-44ae-af90-9f75919bc43c-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.416568 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.416241 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:03.417463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.416770 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.418251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.418228 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "kube-api-access-jmmfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:03.418394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.418379 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:03.418452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.418405 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:03.419943 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.419916 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo" pod "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" (UID: "deacfffc-d5f1-4d0b-865c-ce5b4ae20a46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:03.517091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.516995 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmfw\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-kube-api-access-jmmfw\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.517091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.517032 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.517091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.517045 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.517091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.517054 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.517091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.517062 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.517091 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.517070 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:03.844124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.844083 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:20:03.857318 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.857269 5508 generic.go:356] "Generic (PLEG): container finished" podID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerID="c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd" exitCode=0 Oct 30 22:20:03.857523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.857343 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" Oct 30 22:20:03.857523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.857357 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" event={"ID":"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46","Type":"ContainerDied","Data":"c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd"} Oct 30 22:20:03.857523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.857415 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg" event={"ID":"deacfffc-d5f1-4d0b-865c-ce5b4ae20a46","Type":"ContainerDied","Data":"fcd7dd394555e97ce912708b4644098dbd72440ccd7e8be672239636ebc9d7f4"} Oct 30 22:20:03.857523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.857446 5508 scope.go:122] "RemoveContainer" containerID="c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd" Oct 30 22:20:03.858818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.858785 5508 generic.go:356] "Generic (PLEG): container finished" podID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerID="20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee" exitCode=0 Oct 30 22:20:03.858913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.858828 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" Oct 30 22:20:03.858913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.858868 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" event={"ID":"6043a15c-2b8e-44ae-af90-9f75919bc43c","Type":"ContainerDied","Data":"20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee"} Oct 30 22:20:03.859102 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.858912 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt" event={"ID":"6043a15c-2b8e-44ae-af90-9f75919bc43c","Type":"ContainerDied","Data":"276df4d1b9afe25b01b6becfc1cc0db1ab97ae9e3f25627b819e7669903c6577"} Oct 30 22:20:03.868590 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.868567 5508 scope.go:122] "RemoveContainer" containerID="c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd" Oct 30 22:20:03.868996 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:03.868875 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd\": container with ID starting with c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd not found: ID does not exist" containerID="c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd" Oct 30 22:20:03.868996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.868907 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd"} err="failed to get container status \"c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd\": rpc error: code = NotFound desc = could not find container \"c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd\": container with ID starting with c50df47c10f68e327a1df0ac3b3eea0f96205642650d1c9c35c6b8e7d03671fd not found: ID does not exist" Oct 30 22:20:03.868996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.868924 5508 scope.go:122] "RemoveContainer" containerID="20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee" Oct 30 22:20:03.877656 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.877638 5508 scope.go:122] "RemoveContainer" containerID="20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee" Oct 30 22:20:03.877938 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:03.877917 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee\": container with ID starting with 20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee not found: ID does not exist" containerID="20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee" Oct 30 22:20:03.877997 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:03.877950 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee"} err="failed to get container status \"20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee\": rpc error: code = NotFound desc = could not find container \"20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee\": container with ID starting with 20acefd620ffb9314397ad48c7b8b272a4fe19ff70fa2fb15646b8b65901ecee not found: ID does not exist" Oct 30 22:20:04.019512 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:04.019475 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt"] Oct 30 22:20:04.068633 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:04.068600 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-mpkvt"] Oct 30 22:20:04.177555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:04.177470 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg"] Oct 30 22:20:04.190821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:04.190792 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9dkpg"] Oct 30 22:20:04.365251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:04.365215 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:05.951241 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:05.951210 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" path="/var/lib/kubelet/pods/6043a15c-2b8e-44ae-af90-9f75919bc43c/volumes" Oct 30 22:20:05.951734 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:05.951711 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" path="/var/lib/kubelet/pods/deacfffc-d5f1-4d0b-865c-ce5b4ae20a46/volumes" Oct 30 22:20:06.027458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.027425 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" containerID="cri-o://0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea" gracePeriod=45 Oct 30 22:20:06.077580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.077532 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" containerID="cri-o://08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65" gracePeriod=45 Oct 30 22:20:06.338498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.338477 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:20:06.362726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.362706 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:20:06.448674 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448643 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache\") pod \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " Oct 30 22:20:06.448845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448696 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs\") pod \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " Oct 30 22:20:06.448845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448731 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache\") pod \"afcc5a28-a808-4041-99a3-c27b496b8bb3\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " Oct 30 22:20:06.448845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448752 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt\" (UniqueName: \"kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt\") pod \"afcc5a28-a808-4041-99a3-c27b496b8bb3\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " Oct 30 22:20:06.448845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448775 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm\") pod \"afcc5a28-a808-4041-99a3-c27b496b8bb3\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " Oct 30 22:20:06.448845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448816 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir\") pod \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " Oct 30 22:20:06.448845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448844 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs\") pod \"afcc5a28-a808-4041-99a3-c27b496b8bb3\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448923 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home\") pod \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448958 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir\") pod \"afcc5a28-a808-4041-99a3-c27b496b8bb3\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.448972 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache" pod "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" (UID: "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449001 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home\") pod \"afcc5a28-a808-4041-99a3-c27b496b8bb3\" (UID: \"afcc5a28-a808-4041-99a3-c27b496b8bb3\") " Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449030 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache" pod "afcc5a28-a808-4041-99a3-c27b496b8bb3" (UID: "afcc5a28-a808-4041-99a3-c27b496b8bb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449060 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n\" (UniqueName: \"kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n\") pod \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " Oct 30 22:20:06.449164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449102 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm\") pod \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\" (UID: \"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da\") " Oct 30 22:20:06.449561 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449233 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home" pod "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" (UID: "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.449561 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449350 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home" pod "afcc5a28-a808-4041-99a3-c27b496b8bb3" (UID: "afcc5a28-a808-4041-99a3-c27b496b8bb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.449726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449587 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir" pod "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" (UID: "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.449726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449663 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.449726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449683 5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-home\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.449726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449698 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.449726 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.449712 5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-model-cache\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.450090 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.450051 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir" pod "afcc5a28-a808-4041-99a3-c27b496b8bb3" (UID: "afcc5a28-a808-4041-99a3-c27b496b8bb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.453606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.453561 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt" pod "afcc5a28-a808-4041-99a3-c27b496b8bb3" (UID: "afcc5a28-a808-4041-99a3-c27b496b8bb3"). InnerVolumeSpecName "kube-api-access-hccjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:06.453606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.453562 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm" pod "afcc5a28-a808-4041-99a3-c27b496b8bb3" (UID: "afcc5a28-a808-4041-99a3-c27b496b8bb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.453786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.453615 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm" pod "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" (UID: "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.453786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.453657 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs" pod "afcc5a28-a808-4041-99a3-c27b496b8bb3" (UID: "afcc5a28-a808-4041-99a3-c27b496b8bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:06.453786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.453741 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n" pod "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" (UID: "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da"). InnerVolumeSpecName "kube-api-access-x6j4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:06.456035 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.456004 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs" pod "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" (UID: "5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:06.551057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.550985 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551013 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hccjt\" (UniqueName: \"kubernetes.io/projected/afcc5a28-a808-4041-99a3-c27b496b8bb3-kube-api-access-hccjt\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551024 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551041 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551051 5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc5a28-a808-4041-99a3-c27b496b8bb3-tls-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551060 5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc5a28-a808-4041-99a3-c27b496b8bb3-tmp-dir\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551068 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6j4n\" (UniqueName: \"kubernetes.io/projected/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-kube-api-access-x6j4n\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.551396 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.551077 5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da-dshm\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.877226 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.877195 5508 generic.go:356] "Generic (PLEG): container finished" podID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerID="08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65" exitCode=0 Oct 30 22:20:06.877410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.877237 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" Oct 30 22:20:06.877410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.877280 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" event={"ID":"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da","Type":"ContainerDied","Data":"08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65"} Oct 30 22:20:06.877410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.877341 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh" event={"ID":"5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da","Type":"ContainerDied","Data":"6a744ca99fd9b767148526962ddfb609455e17212904f6442bf4878aa7b1aea0"} Oct 30 22:20:06.877410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.877365 5508 scope.go:122] "RemoveContainer" containerID="08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65" Oct 30 22:20:06.878673 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.878647 5508 generic.go:356] "Generic (PLEG): container finished" podID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerID="0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea" exitCode=0 Oct 30 22:20:06.878780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.878700 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" Oct 30 22:20:06.878780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.878746 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" event={"ID":"afcc5a28-a808-4041-99a3-c27b496b8bb3","Type":"ContainerDied","Data":"0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea"} Oct 30 22:20:06.878873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.878786 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc" event={"ID":"afcc5a28-a808-4041-99a3-c27b496b8bb3","Type":"ContainerDied","Data":"1faab68a45d731d7ec58703290ae0a03675b25ec695e8772ec1d8ac4b9e7fc52"} Oct 30 22:20:06.886348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.886329 5508 scope.go:122] "RemoveContainer" containerID="08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65" Oct 30 22:20:06.886657 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:06.886635 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65\": container with ID starting with 08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65 not found: ID does not exist" containerID="08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65" Oct 30 22:20:06.886723 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.886663 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65"} err="failed to get container status \"08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65\": rpc error: code = NotFound desc = could not find container \"08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65\": container with ID starting with 08e31a88950aac116e6c19832279e7dd638a277135b1b2eab4d6018e2faa9e65 not found: ID does not exist" Oct 30 22:20:06.886723 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.886680 5508 scope.go:122] "RemoveContainer" containerID="0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea" Oct 30 22:20:06.896248 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.896227 5508 scope.go:122] "RemoveContainer" containerID="0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea" Oct 30 22:20:06.896519 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:06.896499 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea\": container with ID starting with 0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea not found: ID does not exist" containerID="0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea" Oct 30 22:20:06.896580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:06.896525 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea"} err="failed to get container status \"0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea\": rpc error: code = NotFound desc = could not find container \"0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea\": container with ID starting with 0fe3dd145fea86da51e0909c57c8c738bd9d3069393a270236a207358edfcbea not found: ID does not exist" Oct 30 22:20:07.039218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:07.039182 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc"] Oct 30 22:20:07.042634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:07.042606 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-2-w6-29cbad-kserve-57694bf49c-vk6xc"] Oct 30 22:20:07.197261 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:07.197175 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh"] Oct 30 22:20:07.200416 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:07.200391 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/distinct-w6-29cbad-kserve-6cd8b769db-wt7vh"] Oct 30 22:20:07.952031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:07.951996 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" path="/var/lib/kubelet/pods/5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da/volumes" Oct 30 22:20:07.952529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:07.952513 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" path="/var/lib/kubelet/pods/afcc5a28-a808-4041-99a3-c27b496b8bb3/volumes" Oct 30 22:20:09.186191 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:09.186152 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:10.372994 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:10.372950 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:20:13.465276 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.465240 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694"] Oct 30 22:20:13.465707 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.465454 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" podUID="57403d72-cdb9-4503-ac30-c0c5d341ee49" containerName="maas-api" containerID="cri-o://6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c" gracePeriod=30 Oct 30 22:20:13.655160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.655124 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:13.724123 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.724066 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 22:20:13.835116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.835080 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca\") pod \"57403d72-cdb9-4503-ac30-c0c5d341ee49\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " Oct 30 22:20:13.835281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.835125 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\") pod \"57403d72-cdb9-4503-ac30-c0c5d341ee49\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " Oct 30 22:20:13.835281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.835196 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg\" (UniqueName: \"kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg\") pod \"57403d72-cdb9-4503-ac30-c0c5d341ee49\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " Oct 30 22:20:13.835406 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.835335 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs\") pod \"57403d72-cdb9-4503-ac30-c0c5d341ee49\" (UID: \"57403d72-cdb9-4503-ac30-c0c5d341ee49\") " Oct 30 22:20:13.835500 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.835475 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca" pod "57403d72-cdb9-4503-ac30-c0c5d341ee49" (UID: "57403d72-cdb9-4503-ac30-c0c5d341ee49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:13.835747 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.835724 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57403d72-cdb9-4503-ac30-c0c5d341ee49-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:13.837513 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.837490 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs" pod "57403d72-cdb9-4503-ac30-c0c5d341ee49" (UID: "57403d72-cdb9-4503-ac30-c0c5d341ee49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:13.837576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.837537 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls" pod "57403d72-cdb9-4503-ac30-c0c5d341ee49" (UID: "57403d72-cdb9-4503-ac30-c0c5d341ee49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:13.837576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.837550 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg" pod "57403d72-cdb9-4503-ac30-c0c5d341ee49" (UID: "57403d72-cdb9-4503-ac30-c0c5d341ee49"). InnerVolumeSpecName "kube-api-access-8ndzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:13.859887 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.859583 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:20:13.920759 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.920721 5508 generic.go:356] "Generic (PLEG): container finished" podID="57403d72-cdb9-4503-ac30-c0c5d341ee49" containerID="6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c" exitCode=0 Oct 30 22:20:13.920962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.920770 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" Oct 30 22:20:13.920962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.920775 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" event={"ID":"57403d72-cdb9-4503-ac30-c0c5d341ee49","Type":"ContainerDied","Data":"6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c"} Oct 30 22:20:13.920962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.920826 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694" event={"ID":"57403d72-cdb9-4503-ac30-c0c5d341ee49","Type":"ContainerDied","Data":"80e5096750a3f2c60ab9eca3bc12cce8cc3f6bf496707cd1edead5ce03bf7e4c"} Oct 30 22:20:13.920962 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.920854 5508 scope.go:122] "RemoveContainer" containerID="6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c" Oct 30 22:20:13.930491 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.930472 5508 scope.go:122] "RemoveContainer" containerID="6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c" Oct 30 22:20:13.930763 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:13.930744 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c\": container with ID starting with 6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c not found: ID does not exist" containerID="6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c" Oct 30 22:20:13.930811 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.930772 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c"} err="failed to get container status \"6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c\": rpc error: code = NotFound desc = could not find container \"6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c\": container with ID starting with 6da53cb127fe5fbd690e1ddc6a21db255951273c42cb077f2f26ecba033b351c not found: ID does not exist" Oct 30 22:20:13.936282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.936262 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ndzg\" (UniqueName: \"kubernetes.io/projected/57403d72-cdb9-4503-ac30-c0c5d341ee49-kube-api-access-8ndzg\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:13.936369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.936286 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:13.936369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:13.936296 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57403d72-cdb9-4503-ac30-c0c5d341ee49-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:14.084178 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:14.084138 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694"] Oct 30 22:20:14.087171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:14.087140 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-29cbad-b6587449d-v8694"] Oct 30 22:20:15.161100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:15.161061 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:15.401490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:15.401458 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:15.575144 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:15.575107 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:15.951066 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:15.950987 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57403d72-cdb9-4503-ac30-c0c5d341ee49" path="/var/lib/kubelet/pods/57403d72-cdb9-4503-ac30-c0c5d341ee49/volumes" Oct 30 22:20:16.224370 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:16.224280 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:20.823433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:20.823378 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:21.528495 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:21.528452 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:21.721914 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:21.721864 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:22.890533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:22.890502 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:23.088063 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.088024 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:23.289375 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.289350 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 22:20:23.334099 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.334063 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume\") pod \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " Oct 30 22:20:23.334271 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.334192 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh\" (UniqueName: \"kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh\") pod \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\" (UID: \"02a354e8-e5ca-4894-9695-e6e14d1b9cfb\") " Oct 30 22:20:23.334521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.334499 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume" pod "02a354e8-e5ca-4894-9695-e6e14d1b9cfb" (UID: "02a354e8-e5ca-4894-9695-e6e14d1b9cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:23.334715 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.334701 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:23.338693 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.338658 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh" pod "02a354e8-e5ca-4894-9695-e6e14d1b9cfb" (UID: "02a354e8-e5ca-4894-9695-e6e14d1b9cfb"). InnerVolumeSpecName "kube-api-access-7tskh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:23.339239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.339216 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:23.348648 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.348625 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt"] Oct 30 22:20:23.348892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.348871 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" containerID="cri-o://6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3" gracePeriod=30 Oct 30 22:20:23.436323 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.436264 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tskh\" (UniqueName: \"kubernetes.io/projected/02a354e8-e5ca-4894-9695-e6e14d1b9cfb-kube-api-access-7tskh\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:23.695075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.695041 5508 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:20:23.695264 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.695098 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:20:23.973127 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.973096 5508 generic.go:356] "Generic (PLEG): container finished" podID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerID="1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4" exitCode=0 Oct 30 22:20:23.973557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.973144 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" event={"ID":"02a354e8-e5ca-4894-9695-e6e14d1b9cfb","Type":"ContainerDied","Data":"1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4"} Oct 30 22:20:23.973557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.973160 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" Oct 30 22:20:23.973557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.973183 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt" event={"ID":"02a354e8-e5ca-4894-9695-e6e14d1b9cfb","Type":"ContainerDied","Data":"fca9c82563c2570914396862e703b55a9ab4747b56d81f249546a97b77b94316"} Oct 30 22:20:23.973557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.973203 5508 scope.go:122] "RemoveContainer" containerID="1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4" Oct 30 22:20:23.981981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.981958 5508 scope.go:122] "RemoveContainer" containerID="1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4" Oct 30 22:20:23.982245 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:23.982227 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4\": container with ID starting with 1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4 not found: ID does not exist" containerID="1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4" Oct 30 22:20:23.982299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:23.982252 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4"} err="failed to get container status \"1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4\": rpc error: code = NotFound desc = could not find container \"1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4\": container with ID starting with 1e5fa4b1c71d0fec0e1a1f0ff94948e904605fa06f65616a259992359687bda4 not found: ID does not exist" Oct 30 22:20:24.149563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:24.149526 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt"] Oct 30 22:20:24.164278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:24.164243 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-29cbad-5c9df696bf-mspdt"] Oct 30 22:20:25.862990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:25.862952 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:25.951817 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:25.951781 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" path="/var/lib/kubelet/pods/02a354e8-e5ca-4894-9695-e6e14d1b9cfb/volumes" Oct 30 22:20:27.317134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:27.317073 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:27.478621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:27.478589 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:28.182551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182519 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92"] Oct 30 22:20:28.182915 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182903 5508 cpu_manager.go:414] "containerMap: removing container" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182916 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182933 5508 cpu_manager.go:414] "containerMap: removing container" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182937 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182945 5508 cpu_manager.go:414] "containerMap: removing container" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182949 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182956 5508 cpu_manager.go:414] "containerMap: removing container" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" Oct 30 22:20:28.182963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182961 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182968 5508 cpu_manager.go:414] "containerMap: removing container" podUID="56303faa-1a27-4e35-992e-99c463aa4829" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182975 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56303faa-1a27-4e35-992e-99c463aa4829" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182983 5508 cpu_manager.go:414] "containerMap: removing container" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182987 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.182996 5508 cpu_manager.go:414] "containerMap: removing container" podUID="6bb0ae00-2005-4a1a-b2d5-738e3718afa1" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183000 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb0ae00-2005-4a1a-b2d5-738e3718afa1" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183005 5508 cpu_manager.go:414] "containerMap: removing container" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183009 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183017 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183021 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183026 5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf5d5ef8-df27-4f0a-bb33-2a64f409480a" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183030 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf5d5ef8-df27-4f0a-bb33-2a64f409480a" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183038 5508 cpu_manager.go:414] "containerMap: removing container" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183042 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183052 5508 cpu_manager.go:414] "containerMap: removing container" podUID="fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183056 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183064 5508 cpu_manager.go:414] "containerMap: removing container" podUID="03672869-8c66-4958-8597-8be4a451981b" containerName="maas-api" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183068 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03672869-8c66-4958-8597-8be4a451981b" containerName="maas-api" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183073 5508 cpu_manager.go:414] "containerMap: removing container" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183077 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183085 5508 cpu_manager.go:414] "containerMap: removing container" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183089 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerName="payload-processing" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183095 5508 cpu_manager.go:414] "containerMap: removing container" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183098 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183103 5508 cpu_manager.go:414] "containerMap: removing container" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183107 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183116 5508 cpu_manager.go:414] "containerMap: removing container" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183121 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183126 5508 cpu_manager.go:414] "containerMap: removing container" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183131 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183137 5508 cpu_manager.go:414] "containerMap: removing container" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183142 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183148 5508 cpu_manager.go:414] "containerMap: removing container" podUID="2ddf0438-70cf-448d-9510-4fe6380d9937" containerName="maas-api" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183153 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddf0438-70cf-448d-9510-4fe6380d9937" containerName="maas-api" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183160 5508 cpu_manager.go:414] "containerMap: removing container" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183164 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183170 5508 cpu_manager.go:414] "containerMap: removing container" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183174 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183181 5508 cpu_manager.go:414] "containerMap: removing container" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" Oct 30 22:20:28.183171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183185 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183192 5508 cpu_manager.go:414] "containerMap: removing container" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183197 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183202 5508 cpu_manager.go:414] "containerMap: removing container" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183206 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beffe2b3-2d29-48b6-99b1-6fd6235c6061" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183214 5508 cpu_manager.go:414] "containerMap: removing container" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183218 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183223 5508 cpu_manager.go:414] "containerMap: removing container" podUID="81b86dbf-0e5b-4c51-bb04-0a6292f15659" containerName="payload-pre-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183228 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b86dbf-0e5b-4c51-bb04-0a6292f15659" containerName="payload-pre-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183235 5508 cpu_manager.go:414] "containerMap: removing container" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183239 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183246 5508 cpu_manager.go:414] "containerMap: removing container" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183250 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183259 5508 cpu_manager.go:414] "containerMap: removing container" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183263 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183270 5508 cpu_manager.go:414] "containerMap: removing container" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183274 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183282 5508 cpu_manager.go:414] "containerMap: removing container" podUID="bac5c48b-8768-4e01-81c2-684c76dab0da" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183286 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac5c48b-8768-4e01-81c2-684c76dab0da" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183294 5508 cpu_manager.go:414] "containerMap: removing container" podUID="57403d72-cdb9-4503-ac30-c0c5d341ee49" containerName="maas-api" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183298 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57403d72-cdb9-4503-ac30-c0c5d341ee49" containerName="maas-api" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183320 5508 cpu_manager.go:414] "containerMap: removing container" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183325 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183336 5508 cpu_manager.go:414] "containerMap: removing container" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerName="payload-pre-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183340 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerName="payload-pre-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183347 5508 cpu_manager.go:414] "containerMap: removing container" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183351 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183357 5508 cpu_manager.go:414] "containerMap: removing container" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183362 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183368 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183372 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183379 5508 cpu_manager.go:414] "containerMap: removing container" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183384 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183443 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa7349f-3fdb-42aa-839c-1d7bc8b1a61b" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183453 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f08e44-69fa-4a74-9f97-5ecaf6606b55" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183460 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb271e6-5de8-433b-bbee-19c54adb327b" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183466 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac5c48b-8768-4e01-81c2-684c76dab0da" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183472 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="57403d72-cdb9-4503-ac30-c0c5d341ee49" containerName="maas-api" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183478 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bf880a-64d5-481b-8cc5-bde91bb82995" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183486 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="444c2438-bb1a-4249-89e9-a14ad8856bdc" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183491 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183497 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d41ced4-4605-46eb-a5bd-33c6ac8b3ecc" containerName="payload-pre-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183503 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9d2e45-3c03-4a9d-aa6c-6f1c3a50e0da" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183509 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183513 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1257b3-9261-404f-955e-65b2aff82ee2" containerName="cleanup" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183520 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53062e4-bdde-401c-a512-465b522fcd64" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183525 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183530 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce63103-453c-4612-b6dd-ae756c80acb2" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183536 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="54eb8069-49be-41e8-a94c-26d8e51710aa" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183541 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="323cc524-025b-4d6d-bf45-845ac6156590" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183548 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="56303faa-1a27-4e35-992e-99c463aa4829" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183553 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6043a15c-2b8e-44ae-af90-9f75919bc43c" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183558 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb0ae00-2005-4a1a-b2d5-738e3718afa1" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183562 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183567 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf5d5ef8-df27-4f0a-bb33-2a64f409480a" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183573 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c637bc-9f13-4e04-b988-a07c04ee163d" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183578 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc69d7f-f37e-4201-a2b4-4344b8d0fc8d" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183584 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="03672869-8c66-4958-8597-8be4a451981b" containerName="maas-api" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183589 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a354e8-e5ca-4894-9695-e6e14d1b9cfb" containerName="payload-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183594 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="34d1011f-485e-4152-af59-fc2f37d02649" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183599 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda608a1-3b58-445c-b3c7-9e75d4044544" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183604 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd33b67-01de-45df-a7af-2920af1ca4ec" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183609 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc09539f-7ac0-4241-b29a-cd19a9cd5f96" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183614 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="deacfffc-d5f1-4d0b-865c-ce5b4ae20a46" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183618 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddf0438-70cf-448d-9510-4fe6380d9937" containerName="maas-api" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183622 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcc5a28-a808-4041-99a3-c27b496b8bb3" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183630 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="676bfdef-67cf-43fb-8c33-2b6a06f6c4f9" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183635 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb72fcb6-dc6a-4fcd-ac79-555c185eaa8d" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183641 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cb7485-ccdd-4c73-81df-45d1511c986b" containerName="istio-proxy" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183646 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb090b2-8a3c-49a3-8c06-13f907d41fa2" containerName="main" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183650 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b86dbf-0e5b-4c51-bb04-0a6292f15659" containerName="payload-pre-processing" Oct 30 22:20:28.184401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.183656 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a33c869-6a90-47f8-87df-3c89dfa0b515" containerName="cleanup" Oct 30 22:20:28.188555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.188537 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287052 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287018 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287215 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287098 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287215 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287160 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsxg\" (UniqueName: \"kubernetes.io/projected/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-kube-api-access-sgsxg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287207 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287293 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287378 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287344 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287538 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287423 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287538 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287498 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.287614 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.287546 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.292898 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.292825 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92"] Oct 30 22:20:28.388170 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388141 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388181 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsxg\" (UniqueName: \"kubernetes.io/projected/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-kube-api-access-sgsxg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388199 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388342 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388394 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388422 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388470 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388502 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388534 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388584 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388696 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.388792 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388739 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.389001 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388886 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.389001 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.388981 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.390914 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.390890 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.391058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.391039 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.446542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.446485 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.446692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.446675 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsxg\" (UniqueName: \"kubernetes.io/projected/889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1-kube-api-access-sgsxg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vmg92\" (UID: \"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.501870 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.501840 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:28.613055 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.612829 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 22:20:28.691606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691580 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.691742 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691620 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.691742 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691661 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.691742 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691717 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.691913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691744 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.691913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691764 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.691913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691826 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.692064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.691969 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.692064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692007 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:28.692161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692086 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs\") pod \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\" (UID: \"0c99a268-d806-4b8f-8aee-824e3a6d6b1a\") " Oct 30 22:20:28.692161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692110 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.692161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692126 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.692488 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692448 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.692601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692476 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.692653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692551 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.692653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692634 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.692750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692651 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.692750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.692667 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.694002 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.693980 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "kube-api-access-rgbnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:28.694170 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.694153 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.694498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.694476 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:28.694555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.694506 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token" pod "0c99a268-d806-4b8f-8aee-824e3a6d6b1a" (UID: "0c99a268-d806-4b8f-8aee-824e3a6d6b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:28.794278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.794198 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.794278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.794231 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbnr\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-kube-api-access-rgbnr\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.794278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.794246 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.794278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.794259 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.794278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.794273 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c99a268-d806-4b8f-8aee-824e3a6d6b1a-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.909154 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.909129 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92"] Oct 30 22:20:28.911991 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:20:28.911965 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889f4f10_f0e9_4c8e_95ff_c2f02d1c04f1.slice/crio-3c7bc44215379968d1f02ce0bca801e16b59a8032cc4043eb29fb4a1285b9af3 WatchSource:0}: Error finding container 3c7bc44215379968d1f02ce0bca801e16b59a8032cc4043eb29fb4a1285b9af3: Status 404 returned error can't find the container with id 3c7bc44215379968d1f02ce0bca801e16b59a8032cc4043eb29fb4a1285b9af3 Oct 30 22:20:28.914219 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.914180 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:28.914358 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.914262 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:28.914435 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:28.914388 5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:20:29.000268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.000232 5508 generic.go:356] "Generic (PLEG): container finished" podID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerID="6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3" exitCode=0 Oct 30 22:20:29.000468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.000294 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" Oct 30 22:20:29.000468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.000332 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" event={"ID":"0c99a268-d806-4b8f-8aee-824e3a6d6b1a","Type":"ContainerDied","Data":"6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3"} Oct 30 22:20:29.000468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.000391 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt" event={"ID":"0c99a268-d806-4b8f-8aee-824e3a6d6b1a","Type":"ContainerDied","Data":"0f06be8954e088b68b89795deda7387202eff3cbe75d9c39188aecd6cccce202"} Oct 30 22:20:29.000468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.000417 5508 scope.go:122] "RemoveContainer" containerID="6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3" Oct 30 22:20:29.002243 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.002220 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" event={"ID":"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1","Type":"ContainerStarted","Data":"6d93efb73aa590e60df9ffbbeea23b3a801cfa546f699199d8341556b972fb30"} Oct 30 22:20:29.002406 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.002268 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" event={"ID":"889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1","Type":"ContainerStarted","Data":"3c7bc44215379968d1f02ce0bca801e16b59a8032cc4043eb29fb4a1285b9af3"} Oct 30 22:20:29.012394 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.012375 5508 scope.go:122] "RemoveContainer" containerID="8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807" Oct 30 22:20:29.024711 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.024691 5508 scope.go:122] "RemoveContainer" containerID="6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3" Oct 30 22:20:29.025171 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:29.025151 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3\": container with ID starting with 6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3 not found: ID does not exist" containerID="6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3" Oct 30 22:20:29.025230 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.025178 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3"} err="failed to get container status \"6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3\": rpc error: code = NotFound desc = could not find container \"6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3\": container with ID starting with 6caa4ffd4367b19b72021e2117b80e554977a4a7b23ae0336dcdb40c54ff70a3 not found: ID does not exist" Oct 30 22:20:29.025230 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.025198 5508 scope.go:122] "RemoveContainer" containerID="8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807" Oct 30 22:20:29.025761 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:29.025480 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807\": container with ID starting with 8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807 not found: ID does not exist" containerID="8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807" Oct 30 22:20:29.025761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.025510 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807"} err="failed to get container status \"8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807\": rpc error: code = NotFound desc = could not find container \"8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807\": container with ID starting with 8082edd6b929e1b267c7cbd165a8ba49e99353fe5a70bcc4b7b71d75e3161807 not found: ID does not exist" Oct 30 22:20:29.174131 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.171463 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podStartSLOduration=1.171445016 podStartE2EDuration="1.171445016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:29.170371887 +0000 UTC m=+2239.359909565" watchObservedRunningTime="2026-10-30 22:20:29.171445016 +0000 UTC m=+2239.360982692" Oct 30 22:20:29.289448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.289411 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt"] Oct 30 22:20:29.297874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.297843 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-65dkt"] Oct 30 22:20:29.363763 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.363729 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:29.502134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.502022 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:29.504197 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.504168 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:29.504345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.504247 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:29.952208 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:29.952171 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" path="/var/lib/kubelet/pods/0c99a268-d806-4b8f-8aee-824e3a6d6b1a/volumes" Oct 30 22:20:30.503146 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:30.503109 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:30.503580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:30.503171 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:30.739228 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:30.739190 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:31.503986 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:31.503952 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:31.504485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:31.504049 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:32.502880 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:32.502838 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:32.503068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:32.502901 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:33.503016 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:33.502973 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:33.503461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:33.503042 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:33.900068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:33.900023 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:34.502966 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:34.502930 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:34.503139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:34.503001 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:35.503113 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:35.503071 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:20:35.503294 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:35.503144 5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:20:35.677810 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:35.677778 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:35.849219 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:35.849185 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:37.218878 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:37.218846 5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:37.220024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:37.219994 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:37.220849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:37.220827 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" Oct 30 22:20:40.373219 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:20:40.373185 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:20:40.592917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:40.592881 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:42.652185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:42.652151 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:42.839238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:42.839201 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:45.574604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:45.574570 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:50.549963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:50.549870 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:55.468851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:55.468796 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:59.134136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:20:59.134090 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:00.416906 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:00.416871 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:04.273357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:04.273322 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:05.808133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:05.808104 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:10.372267 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:21:10.372227 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:21:10.592700 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:10.592669 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:11.191381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:11.191256 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt"] Oct 30 22:21:11.484911 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:11.484829 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:11.652730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:11.652692 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:16.191841 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:16.191782 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerName="payload-pre-processing" containerID="cri-o://68ef0f17df7fcc7517f16ce75fcdd0acfa27a4265cd79fccbf038bf10729d2fd" gracePeriod=25 Oct 30 22:21:18.877445 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:18.877409 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:21:28.878071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:28.878033 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:21:30.260545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.260506 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb"] Oct 30 22:21:30.261011 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.260749 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" podUID="a65189ce-f5c3-4c0b-bf1d-cebae139b17b" containerName="maas-api" containerID="cri-o://f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc" gracePeriod=30 Oct 30 22:21:30.393999 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.393950 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:30.514112 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.514042 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 22:21:30.537506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.537465 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca\") pod \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " Oct 30 22:21:30.537704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.537560 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\") pod \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " Oct 30 22:21:30.537704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.537641 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk\" (UniqueName: \"kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk\") pod \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " Oct 30 22:21:30.537831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.537706 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs\") pod \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\" (UID: \"a65189ce-f5c3-4c0b-bf1d-cebae139b17b\") " Oct 30 22:21:30.537909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.537878 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca" pod "a65189ce-f5c3-4c0b-bf1d-cebae139b17b" (UID: "a65189ce-f5c3-4c0b-bf1d-cebae139b17b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:30.538146 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.538125 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:30.540128 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.540102 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls" pod "a65189ce-f5c3-4c0b-bf1d-cebae139b17b" (UID: "a65189ce-f5c3-4c0b-bf1d-cebae139b17b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:30.540471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.540446 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs" pod "a65189ce-f5c3-4c0b-bf1d-cebae139b17b" (UID: "a65189ce-f5c3-4c0b-bf1d-cebae139b17b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:30.540471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.540454 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk" pod "a65189ce-f5c3-4c0b-bf1d-cebae139b17b" (UID: "a65189ce-f5c3-4c0b-bf1d-cebae139b17b"). InnerVolumeSpecName "kube-api-access-nr8xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:30.640134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.640094 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:30.646289 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.646254 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8xk\" (UniqueName: \"kubernetes.io/projected/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-kube-api-access-nr8xk\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:30.646581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:30.646551 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65189ce-f5c3-4c0b-bf1d-cebae139b17b-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:31.305483 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.305439 5508 generic.go:356] "Generic (PLEG): container finished" podID="a65189ce-f5c3-4c0b-bf1d-cebae139b17b" containerID="f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc" exitCode=0 Oct 30 22:21:31.305998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.305494 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" event={"ID":"a65189ce-f5c3-4c0b-bf1d-cebae139b17b","Type":"ContainerDied","Data":"f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc"} Oct 30 22:21:31.305998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.305501 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" Oct 30 22:21:31.305998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.305548 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb" event={"ID":"a65189ce-f5c3-4c0b-bf1d-cebae139b17b","Type":"ContainerDied","Data":"5f944dad8d4fff88c45403446f8ab173d924e70c2ceba969597f351244cd07ec"} Oct 30 22:21:31.305998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.305590 5508 scope.go:122] "RemoveContainer" containerID="f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc" Oct 30 22:21:31.318771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.318744 5508 scope.go:122] "RemoveContainer" containerID="f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc" Oct 30 22:21:31.319072 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:21:31.319048 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc\": container with ID starting with f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc not found: ID does not exist" containerID="f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc" Oct 30 22:21:31.319157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.319081 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc"} err="failed to get container status \"f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc\": rpc error: code = NotFound desc = could not find container \"f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc\": container with ID starting with f5f1d2ae1108f108c74035eb64a2c7863121f23505eb0dc9e0339c06f47269dc not found: ID does not exist" Oct 30 22:21:31.467777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.467742 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb"] Oct 30 22:21:31.518177 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.518136 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-0f42cb-776fdcb55-hgdfb"] Oct 30 22:21:31.919049 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.919015 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:31.950874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:31.950844 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65189ce-f5c3-4c0b-bf1d-cebae139b17b" path="/var/lib/kubelet/pods/a65189ce-f5c3-4c0b-bf1d-cebae139b17b/volumes" Oct 30 22:21:32.188254 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:32.188167 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:32.870562 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:32.870532 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:34.249650 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:34.249616 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:34.446692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:34.446646 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:36.332645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.332613 5508 generic.go:356] "Generic (PLEG): container finished" podID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerID="68ef0f17df7fcc7517f16ce75fcdd0acfa27a4265cd79fccbf038bf10729d2fd" exitCode=0 Oct 30 22:21:36.332967 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.332690 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" event={"ID":"d9f665cb-704b-49b8-833f-3015ad15c1cb","Type":"ContainerDied","Data":"68ef0f17df7fcc7517f16ce75fcdd0acfa27a4265cd79fccbf038bf10729d2fd"} Oct 30 22:21:36.440636 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.440605 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 22:21:36.496168 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.496099 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume\") pod \"d9f665cb-704b-49b8-833f-3015ad15c1cb\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " Oct 30 22:21:36.496168 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.496144 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt\" (UniqueName: \"kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt\") pod \"d9f665cb-704b-49b8-833f-3015ad15c1cb\" (UID: \"d9f665cb-704b-49b8-833f-3015ad15c1cb\") " Oct 30 22:21:36.496545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.496523 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume" pod "d9f665cb-704b-49b8-833f-3015ad15c1cb" (UID: "d9f665cb-704b-49b8-833f-3015ad15c1cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:36.498381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.498362 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt" pod "d9f665cb-704b-49b8-833f-3015ad15c1cb" (UID: "d9f665cb-704b-49b8-833f-3015ad15c1cb"). InnerVolumeSpecName "kube-api-access-445vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:36.598038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.598000 5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f665cb-704b-49b8-833f-3015ad15c1cb-plugins-config-volume\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:36.598038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:36.598031 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445vt\" (UniqueName: \"kubernetes.io/projected/d9f665cb-704b-49b8-833f-3015ad15c1cb-kube-api-access-445vt\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:37.345478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:37.345430 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" event={"ID":"d9f665cb-704b-49b8-833f-3015ad15c1cb","Type":"ContainerDied","Data":"3a162dfe5ac3c9e9eb7d6142f934df85eb0418f280c6a26eda72e123ab235f4d"} Oct 30 22:21:37.345478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:37.345462 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt" Oct 30 22:21:37.345990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:37.345537 5508 scope.go:122] "RemoveContainer" containerID="68ef0f17df7fcc7517f16ce75fcdd0acfa27a4265cd79fccbf038bf10729d2fd" Oct 30 22:21:37.508148 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:37.508115 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt"] Oct 30 22:21:37.557404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:37.557365 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-0f42cb-9c848df75-8fpjt"] Oct 30 22:21:37.951765 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:37.951734 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" path="/var/lib/kubelet/pods/d9f665cb-704b-49b8-833f-3015ad15c1cb/volumes" Oct 30 22:21:39.168955 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:39.168914 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:39.338150 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:39.338118 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:39.497854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:39.497770 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg"] Oct 30 22:21:39.498044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:39.498013 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" containerID="cri-o://4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487" gracePeriod=30 Oct 30 22:21:39.526084 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:39.526056 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:40.380127 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:21:40.380090 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:21:40.848335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:40.848275 5508 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:40.848540 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:40.848370 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:40.988956 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:40.988924 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:41.165231 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:41.165150 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:41.915542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:41.915505 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:44.750183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.750158 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:21:44.895769 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.895743 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.895935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.895787 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.895935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.895834 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.895935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.895864 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.895935 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.895919 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.896162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.895993 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.896162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896017 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.896162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896080 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.896162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896135 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket\") pod \"d6976822-f864-4d39-8902-576c8e167e82\" (UID: \"d6976822-f864-4d39-8902-576c8e167e82\") " Oct 30 22:21:44.896162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896141 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.896442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896177 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.896442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896267 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.896554 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896505 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.896837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896723 5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-data\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.896837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896748 5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.896837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896764 5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-credential-socket\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.896837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896779 5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-workload-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.897154 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.896898 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:44.898275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.898253 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:44.898400 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.898286 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.898488 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.898463 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "kube-api-access-slfcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:44.898535 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.898516 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo" pod "d6976822-f864-4d39-8902-576c8e167e82" (UID: "d6976822-f864-4d39-8902-576c8e167e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:44.998080 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.998049 5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6976822-f864-4d39-8902-576c8e167e82-istiod-ca-cert\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.998080 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.998077 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfcw\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-kube-api-access-slfcw\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.998080 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.998089 5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6976822-f864-4d39-8902-576c8e167e82-istio-podinfo\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.998392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.998101 5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6976822-f864-4d39-8902-576c8e167e82-istio-envoy\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.998392 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:44.998115 5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6976822-f864-4d39-8902-576c8e167e82-istio-token\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.385920 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.385882 5508 generic.go:356] "Generic (PLEG): container finished" podID="d6976822-f864-4d39-8902-576c8e167e82" containerID="4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487" exitCode=0 Oct 30 22:21:45.386139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.385936 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" Oct 30 22:21:45.386139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.385968 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" event={"ID":"d6976822-f864-4d39-8902-576c8e167e82","Type":"ContainerDied","Data":"4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487"} Oct 30 22:21:45.386139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.386028 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg" event={"ID":"d6976822-f864-4d39-8902-576c8e167e82","Type":"ContainerDied","Data":"50eb24ea420857be7913d139834f78a2602590715a22da8cd222e20593e69a95"} Oct 30 22:21:45.386139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.386050 5508 scope.go:122] "RemoveContainer" containerID="4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487" Oct 30 22:21:45.395537 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.395518 5508 scope.go:122] "RemoveContainer" containerID="4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487" Oct 30 22:21:45.395815 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:21:45.395793 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487\": container with ID starting with 4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487 not found: ID does not exist" containerID="4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487" Oct 30 22:21:45.395864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.395825 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487"} err="failed to get container status \"4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487\": rpc error: code = NotFound desc = could not find container \"4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487\": container with ID starting with 4ed121258c1dd5359499a5422d414699290f6b0323b37f5aa2d3ba06191aa487 not found: ID does not exist" Oct 30 22:21:45.547982 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.547951 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg"] Oct 30 22:21:45.597046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.597012 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-kt4jg"] Oct 30 22:21:45.951075 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:45.951045 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6976822-f864-4d39-8902-576c8e167e82" path="/var/lib/kubelet/pods/d6976822-f864-4d39-8902-576c8e167e82/volumes" Oct 30 22:21:47.641543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:21:47.641495 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:04.122076 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:04.122038 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:06.642755 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.642718 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c"] Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643095 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643106 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643125 5508 cpu_manager.go:414] "containerMap: removing container" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerName="payload-pre-processing" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643130 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerName="payload-pre-processing" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643145 5508 cpu_manager.go:414] "containerMap: removing container" podUID="a65189ce-f5c3-4c0b-bf1d-cebae139b17b" containerName="maas-api" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643150 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65189ce-f5c3-4c0b-bf1d-cebae139b17b" containerName="maas-api" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643158 5508 cpu_manager.go:414] "containerMap: removing container" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643162 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643171 5508 cpu_manager.go:414] "containerMap: removing container" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" Oct 30 22:22:06.643176 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643177 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643185 5508 cpu_manager.go:414] "containerMap: removing container" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643190 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643251 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f665cb-704b-49b8-833f-3015ad15c1cb" containerName="payload-pre-processing" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643261 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65189ce-f5c3-4c0b-bf1d-cebae139b17b" containerName="maas-api" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643267 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643274 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6976822-f864-4d39-8902-576c8e167e82" containerName="istio-proxy" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643279 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f9c240d-6eca-4f39-8f16-0ce7a40cf274" containerName="istio-proxy" Oct 30 22:22:06.643570 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.643285 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="86590685-ff39-41c3-a263-b54d75714605" containerName="istio-proxy" Oct 30 22:22:06.646183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.646166 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.699105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.699077 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:22:06.699105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.699090 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:22:06.736854 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.736820 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.737052 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.736873 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.737052 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.736905 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfr7\" (UniqueName: \"kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.748650 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.748619 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:22:06.748650 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.748641 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:22:06.750067 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.750048 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c"] Oct 30 22:22:06.837874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.837839 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.838071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.837898 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.838071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.837945 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfr7\" (UniqueName: \"kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.838626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.838601 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.840384 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.840360 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.895504 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.895445 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfr7\" (UniqueName: \"kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7\") pod \"maas-api-revoke-sub-85237e45-c7l5c\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:06.958500 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:06.958467 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:07.332079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:07.331924 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c"] Oct 30 22:22:07.335113 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:22:07.335081 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5607ad_bd58_439a_9e5c_e70ff5608a04.slice/crio-727d5a943bd293589c325b1e26921a468b7912aea5bc6b498a0fd0a2a0c878d0 WatchSource:0}: Error finding container 727d5a943bd293589c325b1e26921a468b7912aea5bc6b498a0fd0a2a0c878d0: Status 404 returned error can't find the container with id 727d5a943bd293589c325b1e26921a468b7912aea5bc6b498a0fd0a2a0c878d0 Oct 30 22:22:07.507684 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:07.507655 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" event={"ID":"ac5607ad-bd58-439a-9e5c-e70ff5608a04","Type":"ContainerStarted","Data":"727d5a943bd293589c325b1e26921a468b7912aea5bc6b498a0fd0a2a0c878d0"} Oct 30 22:22:08.514438 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:08.514344 5508 generic.go:356] "Generic (PLEG): container finished" podID="ac5607ad-bd58-439a-9e5c-e70ff5608a04" containerID="0664525dad338afa5de3e2da248b3ce8590c78134aa6c4fbffa716e6149c4b0a" exitCode=0 Oct 30 22:22:08.514864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:08.514427 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" event={"ID":"ac5607ad-bd58-439a-9e5c-e70ff5608a04","Type":"ContainerDied","Data":"0664525dad338afa5de3e2da248b3ce8590c78134aa6c4fbffa716e6149c4b0a"} Oct 30 22:22:09.657864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.657836 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:09.665046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.665022 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca\") pod \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " Oct 30 22:22:09.665225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.665111 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7\" (UniqueName: \"kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7\") pod \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " Oct 30 22:22:09.665225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.665205 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca\") pod \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\" (UID: \"ac5607ad-bd58-439a-9e5c-e70ff5608a04\") " Oct 30 22:22:09.665555 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.665523 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca" pod "ac5607ad-bd58-439a-9e5c-e70ff5608a04" (UID: "ac5607ad-bd58-439a-9e5c-e70ff5608a04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:22:09.667193 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.667176 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca" pod "ac5607ad-bd58-439a-9e5c-e70ff5608a04" (UID: "ac5607ad-bd58-439a-9e5c-e70ff5608a04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:22:09.667262 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.667232 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7" pod "ac5607ad-bd58-439a-9e5c-e70ff5608a04" (UID: "ac5607ad-bd58-439a-9e5c-e70ff5608a04"). InnerVolumeSpecName "kube-api-access-cmfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:22:09.766891 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.766865 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:09.766891 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.766889 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfr7\" (UniqueName: \"kubernetes.io/projected/ac5607ad-bd58-439a-9e5c-e70ff5608a04-kube-api-access-cmfr7\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:09.767056 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:09.766900 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5607ad-bd58-439a-9e5c-e70ff5608a04-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:22:10.372566 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:22:10.372530 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:22:10.525062 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:10.525031 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" event={"ID":"ac5607ad-bd58-439a-9e5c-e70ff5608a04","Type":"ContainerDied","Data":"727d5a943bd293589c325b1e26921a468b7912aea5bc6b498a0fd0a2a0c878d0"} Oct 30 22:22:10.525223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:10.525072 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c" Oct 30 22:22:10.525223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:10.525080 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727d5a943bd293589c325b1e26921a468b7912aea5bc6b498a0fd0a2a0c878d0" Oct 30 22:22:19.135357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:19.135315 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:21.707689 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:21.707644 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:30.060712 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:30.060674 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:30.247995 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:30.247960 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:34.584797 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:22:34.584760 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:40.372751 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:22:40.372714 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:23:01.856733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:01.856687 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:02.026904 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:02.026865 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:05.155369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:05.155335 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:05.454988 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:05.454916 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:10.371652 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:23:10.371605 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:23:15.602377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.602340 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf"] Oct 30 22:23:15.602855 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.602812 5508 cpu_manager.go:414] "containerMap: removing container" podUID="ac5607ad-bd58-439a-9e5c-e70ff5608a04" containerName="revoke-keys" Oct 30 22:23:15.602855 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.602826 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5607ad-bd58-439a-9e5c-e70ff5608a04" containerName="revoke-keys" Oct 30 22:23:15.602974 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.602892 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5607ad-bd58-439a-9e5c-e70ff5608a04" containerName="revoke-keys" Oct 30 22:23:15.602974 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.602904 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c99a268-d806-4b8f-8aee-824e3a6d6b1a" containerName="istio-proxy" Oct 30 22:23:15.606206 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.606186 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.707939 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.707904 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:15.707939 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.707932 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:15.708135 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.707956 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:23:15.708135 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.707956 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:15.709503 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.709484 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf"] Oct 30 22:23:15.718481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.718464 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.718579 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.718494 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdpq\" (UniqueName: \"kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.718579 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.718514 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.819830 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.819794 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.819830 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.819837 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdpq\" (UniqueName: \"kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.820058 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.819854 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.820558 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.820535 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.822429 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.822412 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.877639 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.877570 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdpq\" (UniqueName: \"kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq\") pod \"maas-api-revoke-sub-c1e271a9-slkhf\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:15.918715 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:15.918691 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:16.292856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:16.292663 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf"] Oct 30 22:23:16.295641 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:23:16.295606 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f006a18_c91f_49e6_9759_54b8aaf008f2.slice/crio-970bcf537ec0ed2069571c2b6cb938cbcb6cad71d5511a3ed39ba2ac1427c6bc WatchSource:0}: Error finding container 970bcf537ec0ed2069571c2b6cb938cbcb6cad71d5511a3ed39ba2ac1427c6bc: Status 404 returned error can't find the container with id 970bcf537ec0ed2069571c2b6cb938cbcb6cad71d5511a3ed39ba2ac1427c6bc Oct 30 22:23:16.854462 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:16.854368 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" event={"ID":"7f006a18-c91f-49e6-9759-54b8aaf008f2","Type":"ContainerStarted","Data":"ccd0ca1cc97db8fab5a885cf92008ecf6e09cd2fb2834b4f2127648d40eefe64"} Oct 30 22:23:16.854462 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:16.854453 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" event={"ID":"7f006a18-c91f-49e6-9759-54b8aaf008f2","Type":"ContainerStarted","Data":"970bcf537ec0ed2069571c2b6cb938cbcb6cad71d5511a3ed39ba2ac1427c6bc"} Oct 30 22:23:16.970412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:16.970353 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" podStartSLOduration=1.970339316 podStartE2EDuration="1.970339316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:16.968620344 +0000 UTC m=+2407.158158019" watchObservedRunningTime="2026-10-30 22:23:16.970339316 +0000 UTC m=+2407.159876969" Oct 30 22:23:17.860622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:17.860582 5508 generic.go:356] "Generic (PLEG): container finished" podID="7f006a18-c91f-49e6-9759-54b8aaf008f2" containerID="ccd0ca1cc97db8fab5a885cf92008ecf6e09cd2fb2834b4f2127648d40eefe64" exitCode=0 Oct 30 22:23:17.861044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:17.860664 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" event={"ID":"7f006a18-c91f-49e6-9759-54b8aaf008f2","Type":"ContainerDied","Data":"ccd0ca1cc97db8fab5a885cf92008ecf6e09cd2fb2834b4f2127648d40eefe64"} Oct 30 22:23:19.004818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.004791 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:19.051945 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.051916 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca\") pod \"7f006a18-c91f-49e6-9759-54b8aaf008f2\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " Oct 30 22:23:19.052078 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.051966 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq\" (UniqueName: \"kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq\") pod \"7f006a18-c91f-49e6-9759-54b8aaf008f2\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " Oct 30 22:23:19.052078 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.052012 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca\") pod \"7f006a18-c91f-49e6-9759-54b8aaf008f2\" (UID: \"7f006a18-c91f-49e6-9759-54b8aaf008f2\") " Oct 30 22:23:19.052257 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.052233 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca" pod "7f006a18-c91f-49e6-9759-54b8aaf008f2" (UID: "7f006a18-c91f-49e6-9759-54b8aaf008f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:19.052423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.052405 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.054267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.054247 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca" pod "7f006a18-c91f-49e6-9759-54b8aaf008f2" (UID: "7f006a18-c91f-49e6-9759-54b8aaf008f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:19.054352 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.054297 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq" pod "7f006a18-c91f-49e6-9759-54b8aaf008f2" (UID: "7f006a18-c91f-49e6-9759-54b8aaf008f2"). InnerVolumeSpecName "kube-api-access-zgdpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:19.153545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.153478 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdpq\" (UniqueName: \"kubernetes.io/projected/7f006a18-c91f-49e6-9759-54b8aaf008f2-kube-api-access-zgdpq\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.153545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.153504 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f006a18-c91f-49e6-9759-54b8aaf008f2-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.870874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.870839 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" Oct 30 22:23:19.871049 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.870873 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf" event={"ID":"7f006a18-c91f-49e6-9759-54b8aaf008f2","Type":"ContainerDied","Data":"970bcf537ec0ed2069571c2b6cb938cbcb6cad71d5511a3ed39ba2ac1427c6bc"} Oct 30 22:23:19.871049 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:19.870924 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="970bcf537ec0ed2069571c2b6cb938cbcb6cad71d5511a3ed39ba2ac1427c6bc" Oct 30 22:23:20.824735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:20.824705 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:21.004548 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:21.004517 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:35.029330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:35.025381 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:40.371819 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:23:40.371785 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:23:46.996093 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:46.996057 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt"] Oct 30 22:23:46.996585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:46.996480 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7f006a18-c91f-49e6-9759-54b8aaf008f2" containerName="revoke-keys" Oct 30 22:23:46.996585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:46.996492 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f006a18-c91f-49e6-9759-54b8aaf008f2" containerName="revoke-keys" Oct 30 22:23:46.996585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:46.996564 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f006a18-c91f-49e6-9759-54b8aaf008f2" containerName="revoke-keys" Oct 30 22:23:47.001100 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.001072 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.055035 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.055011 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:47.055155 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.055010 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:47.065620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.065595 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.065743 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.065630 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhvh\" (UniqueName: \"kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.065816 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.065789 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.104506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.104480 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:23:47.104506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.104482 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:47.106020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.106000 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt"] Oct 30 22:23:47.167213 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.167187 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.167374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.167223 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhvh\" (UniqueName: \"kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.167374 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.167358 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.167899 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.167874 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.169967 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.169941 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.231653 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.231617 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhvh\" (UniqueName: \"kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh\") pod \"maas-api-revoke-sub-2e1f5b5f-7sbdt\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.316516 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.316435 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:47.693319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:47.693278 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt"] Oct 30 22:23:47.696177 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:23:47.696147 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a3caea_974d_4618_86ec_74a237e3cc5e.slice/crio-38a11ea89f9475f1ec803e10fc28a8e5b67cacc0def343ecf803b7490b1ca437 WatchSource:0}: Error finding container 38a11ea89f9475f1ec803e10fc28a8e5b67cacc0def343ecf803b7490b1ca437: Status 404 returned error can't find the container with id 38a11ea89f9475f1ec803e10fc28a8e5b67cacc0def343ecf803b7490b1ca437 Oct 30 22:23:48.011844 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:48.011812 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" event={"ID":"d4a3caea-974d-4618-86ec-74a237e3cc5e","Type":"ContainerStarted","Data":"6b43ab779d6546cee3cac5b137543d5b507bf2a39bdadbf69a26b8f6e196ccca"} Oct 30 22:23:48.012192 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:48.011868 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" event={"ID":"d4a3caea-974d-4618-86ec-74a237e3cc5e","Type":"ContainerStarted","Data":"38a11ea89f9475f1ec803e10fc28a8e5b67cacc0def343ecf803b7490b1ca437"} Oct 30 22:23:48.137951 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:48.137626 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" podStartSLOduration=2.137609702 podStartE2EDuration="2.137609702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:48.136180803 +0000 UTC m=+2438.325718484" watchObservedRunningTime="2026-10-30 22:23:48.137609702 +0000 UTC m=+2438.327147378" Oct 30 22:23:49.018071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:49.017985 5508 generic.go:356] "Generic (PLEG): container finished" podID="d4a3caea-974d-4618-86ec-74a237e3cc5e" containerID="6b43ab779d6546cee3cac5b137543d5b507bf2a39bdadbf69a26b8f6e196ccca" exitCode=0 Oct 30 22:23:49.018071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:49.018030 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" event={"ID":"d4a3caea-974d-4618-86ec-74a237e3cc5e","Type":"ContainerDied","Data":"6b43ab779d6546cee3cac5b137543d5b507bf2a39bdadbf69a26b8f6e196ccca"} Oct 30 22:23:50.163928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.163903 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:50.298627 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.298548 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca\") pod \"d4a3caea-974d-4618-86ec-74a237e3cc5e\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " Oct 30 22:23:50.298777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.298625 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca\") pod \"d4a3caea-974d-4618-86ec-74a237e3cc5e\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " Oct 30 22:23:50.298777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.298729 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh\" (UniqueName: \"kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh\") pod \"d4a3caea-974d-4618-86ec-74a237e3cc5e\" (UID: \"d4a3caea-974d-4618-86ec-74a237e3cc5e\") " Oct 30 22:23:50.298982 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.298962 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca" pod "d4a3caea-974d-4618-86ec-74a237e3cc5e" (UID: "d4a3caea-974d-4618-86ec-74a237e3cc5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:50.299336 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.299256 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.300939 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.300914 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca" pod "d4a3caea-974d-4618-86ec-74a237e3cc5e" (UID: "d4a3caea-974d-4618-86ec-74a237e3cc5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:50.301035 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.300944 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh" pod "d4a3caea-974d-4618-86ec-74a237e3cc5e" (UID: "d4a3caea-974d-4618-86ec-74a237e3cc5e"). InnerVolumeSpecName "kube-api-access-9qhvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:50.400183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.400154 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a3caea-974d-4618-86ec-74a237e3cc5e-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:50.400183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:50.400182 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhvh\" (UniqueName: \"kubernetes.io/projected/d4a3caea-974d-4618-86ec-74a237e3cc5e-kube-api-access-9qhvh\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:51.029077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:51.029043 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" event={"ID":"d4a3caea-974d-4618-86ec-74a237e3cc5e","Type":"ContainerDied","Data":"38a11ea89f9475f1ec803e10fc28a8e5b67cacc0def343ecf803b7490b1ca437"} Oct 30 22:23:51.029243 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:51.029084 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt" Oct 30 22:23:51.029243 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:51.029090 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38a11ea89f9475f1ec803e10fc28a8e5b67cacc0def343ecf803b7490b1ca437" Oct 30 22:23:59.461609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.461571 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz"] Oct 30 22:23:59.462170 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.462149 5508 cpu_manager.go:414] "containerMap: removing container" podUID="d4a3caea-974d-4618-86ec-74a237e3cc5e" containerName="revoke-keys" Oct 30 22:23:59.462250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.462169 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a3caea-974d-4618-86ec-74a237e3cc5e" containerName="revoke-keys" Oct 30 22:23:59.462336 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.462289 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a3caea-974d-4618-86ec-74a237e3cc5e" containerName="revoke-keys" Oct 30 22:23:59.465542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.465520 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.517468 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.517439 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:23:59.566636 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.566601 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:23:59.566636 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.566619 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:23:59.566841 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.566680 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:23:59.568205 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.568186 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz"] Oct 30 22:23:59.586767 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.586738 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.586902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.586779 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.586902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.586879 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjd9\" (UniqueName: \"kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.688269 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.688236 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.688489 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.688288 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.688489 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.688368 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjd9\" (UniqueName: \"kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.689034 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.689010 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.690611 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.690585 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.744797 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.744743 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjd9\" (UniqueName: \"kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9\") pod \"maas-api-revoke-sub-ec843018-g9lxz\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:23:59.778651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:23:59.778625 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:24:00.151740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:00.151707 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz"] Oct 30 22:24:00.154932 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:24:00.154905 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61eab7e2_895d_4d0a_966e_20b860613890.slice/crio-7bfc1bd81851939ec8d98023a1abb98aa25da0ae12045aad92640d02a8e55e08 WatchSource:0}: Error finding container 7bfc1bd81851939ec8d98023a1abb98aa25da0ae12045aad92640d02a8e55e08: Status 404 returned error can't find the container with id 7bfc1bd81851939ec8d98023a1abb98aa25da0ae12045aad92640d02a8e55e08 Oct 30 22:24:01.075976 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:01.075950 5508 generic.go:356] "Generic (PLEG): container finished" podID="61eab7e2-895d-4d0a-966e-20b860613890" containerID="12417291024ef1ef6951767d8e5f7c13dc81f649f8283b32b1faf95ad2ac842f" exitCode=0 Oct 30 22:24:01.076293 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:01.075999 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" event={"ID":"61eab7e2-895d-4d0a-966e-20b860613890","Type":"ContainerDied","Data":"12417291024ef1ef6951767d8e5f7c13dc81f649f8283b32b1faf95ad2ac842f"} Oct 30 22:24:01.076293 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:01.076035 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" event={"ID":"61eab7e2-895d-4d0a-966e-20b860613890","Type":"ContainerStarted","Data":"7bfc1bd81851939ec8d98023a1abb98aa25da0ae12045aad92640d02a8e55e08"} Oct 30 22:24:02.218132 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.218101 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:24:02.312111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.312074 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca\") pod \"61eab7e2-895d-4d0a-966e-20b860613890\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " Oct 30 22:24:02.312358 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.312207 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9\" (UniqueName: \"kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9\") pod \"61eab7e2-895d-4d0a-966e-20b860613890\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " Oct 30 22:24:02.312358 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.312231 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca\") pod \"61eab7e2-895d-4d0a-966e-20b860613890\" (UID: \"61eab7e2-895d-4d0a-966e-20b860613890\") " Oct 30 22:24:02.312690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.312661 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca" pod "61eab7e2-895d-4d0a-966e-20b860613890" (UID: "61eab7e2-895d-4d0a-966e-20b860613890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:02.314554 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.314521 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca" pod "61eab7e2-895d-4d0a-966e-20b860613890" (UID: "61eab7e2-895d-4d0a-966e-20b860613890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:02.314680 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.314644 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9" pod "61eab7e2-895d-4d0a-966e-20b860613890" (UID: "61eab7e2-895d-4d0a-966e-20b860613890"). InnerVolumeSpecName "kube-api-access-vfjd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:02.413399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.413288 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:02.413399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.413353 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjd9\" (UniqueName: \"kubernetes.io/projected/61eab7e2-895d-4d0a-966e-20b860613890-kube-api-access-vfjd9\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:02.413399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:02.413366 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61eab7e2-895d-4d0a-966e-20b860613890-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:03.087689 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:03.087650 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" event={"ID":"61eab7e2-895d-4d0a-966e-20b860613890","Type":"ContainerDied","Data":"7bfc1bd81851939ec8d98023a1abb98aa25da0ae12045aad92640d02a8e55e08"} Oct 30 22:24:03.087689 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:03.087690 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz" Oct 30 22:24:03.087917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:03.087703 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bfc1bd81851939ec8d98023a1abb98aa25da0ae12045aad92640d02a8e55e08" Oct 30 22:24:07.013575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:07.013521 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:07.331743 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:07.331710 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:10.372959 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:24:10.372922 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:24:34.423546 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.423513 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:34.881171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.881136 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv"] Oct 30 22:24:34.881560 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.881547 5508 cpu_manager.go:414] "containerMap: removing container" podUID="61eab7e2-895d-4d0a-966e-20b860613890" containerName="revoke-keys" Oct 30 22:24:34.881612 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.881561 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61eab7e2-895d-4d0a-966e-20b860613890" containerName="revoke-keys" Oct 30 22:24:34.881647 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.881632 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="61eab7e2-895d-4d0a-966e-20b860613890" containerName="revoke-keys" Oct 30 22:24:34.884536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.884520 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.895906 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.895878 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7pg\" (UniqueName: \"kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.896079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.895911 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.896079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.895944 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.938758 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.938726 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:34.938931 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.938769 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:34.938931 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.938773 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:34.988180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.988145 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:24:34.989988 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.989967 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv"] Oct 30 22:24:34.996583 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.996558 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7pg\" (UniqueName: \"kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.996692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.996592 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.996692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.996628 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.997332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.997293 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:34.999169 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:34.999145 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:35.055991 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:35.055960 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7pg\" (UniqueName: \"kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg\") pod \"maas-api-revoke-sub-046e1343-7lrqv\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:35.198122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:35.198043 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:35.326927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:35.326895 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:35.573359 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:35.573333 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv"] Oct 30 22:24:35.575913 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:24:35.575884 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59783c93_2c17_45c0_accd_18cf0c8962b5.slice/crio-29173519c35f64710e60aea6bd92e5e85da18213150ff78dcc8996aca9060beb WatchSource:0}: Error finding container 29173519c35f64710e60aea6bd92e5e85da18213150ff78dcc8996aca9060beb: Status 404 returned error can't find the container with id 29173519c35f64710e60aea6bd92e5e85da18213150ff78dcc8996aca9060beb Oct 30 22:24:36.246356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:36.246234 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" event={"ID":"59783c93-2c17-45c0-accd-18cf0c8962b5","Type":"ContainerStarted","Data":"440e6a8d1db5b3c1d2570d98d28543459c2bf6daa7dc55af320f0a6ef6592aff"} Oct 30 22:24:36.246356 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:36.246295 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" event={"ID":"59783c93-2c17-45c0-accd-18cf0c8962b5","Type":"ContainerStarted","Data":"29173519c35f64710e60aea6bd92e5e85da18213150ff78dcc8996aca9060beb"} Oct 30 22:24:36.362184 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:36.362131 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" podStartSLOduration=2.362114552 podStartE2EDuration="2.362114552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:36.361220019 +0000 UTC m=+2486.550757710" watchObservedRunningTime="2026-10-30 22:24:36.362114552 +0000 UTC m=+2486.551652229" Oct 30 22:24:37.252321 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:37.252272 5508 generic.go:356] "Generic (PLEG): container finished" podID="59783c93-2c17-45c0-accd-18cf0c8962b5" containerID="440e6a8d1db5b3c1d2570d98d28543459c2bf6daa7dc55af320f0a6ef6592aff" exitCode=0 Oct 30 22:24:37.252735 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:37.252369 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" event={"ID":"59783c93-2c17-45c0-accd-18cf0c8962b5","Type":"ContainerDied","Data":"440e6a8d1db5b3c1d2570d98d28543459c2bf6daa7dc55af320f0a6ef6592aff"} Oct 30 22:24:38.388604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.388577 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:38.429061 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.429030 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca\") pod \"59783c93-2c17-45c0-accd-18cf0c8962b5\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " Oct 30 22:24:38.429173 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.429142 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg\" (UniqueName: \"kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg\") pod \"59783c93-2c17-45c0-accd-18cf0c8962b5\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " Oct 30 22:24:38.429218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.429194 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca\") pod \"59783c93-2c17-45c0-accd-18cf0c8962b5\" (UID: \"59783c93-2c17-45c0-accd-18cf0c8962b5\") " Oct 30 22:24:38.429381 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.429361 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca" pod "59783c93-2c17-45c0-accd-18cf0c8962b5" (UID: "59783c93-2c17-45c0-accd-18cf0c8962b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:38.429505 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.429486 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:38.431415 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.431392 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca" pod "59783c93-2c17-45c0-accd-18cf0c8962b5" (UID: "59783c93-2c17-45c0-accd-18cf0c8962b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:38.431539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.431516 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg" pod "59783c93-2c17-45c0-accd-18cf0c8962b5" (UID: "59783c93-2c17-45c0-accd-18cf0c8962b5"). InnerVolumeSpecName "kube-api-access-ml7pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:38.530104 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.530020 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml7pg\" (UniqueName: \"kubernetes.io/projected/59783c93-2c17-45c0-accd-18cf0c8962b5-kube-api-access-ml7pg\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:38.530104 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:38.530053 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59783c93-2c17-45c0-accd-18cf0c8962b5-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:39.264450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:39.264414 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" event={"ID":"59783c93-2c17-45c0-accd-18cf0c8962b5","Type":"ContainerDied","Data":"29173519c35f64710e60aea6bd92e5e85da18213150ff78dcc8996aca9060beb"} Oct 30 22:24:39.264450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:39.264445 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv" Oct 30 22:24:39.264710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:39.264493 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29173519c35f64710e60aea6bd92e5e85da18213150ff78dcc8996aca9060beb" Oct 30 22:24:40.371733 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:24:40.371697 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:24:45.613321 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.613277 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr"] Oct 30 22:24:45.613892 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.613871 5508 cpu_manager.go:414] "containerMap: removing container" podUID="59783c93-2c17-45c0-accd-18cf0c8962b5" containerName="revoke-keys" Oct 30 22:24:45.613979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.613894 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59783c93-2c17-45c0-accd-18cf0c8962b5" containerName="revoke-keys" Oct 30 22:24:45.614030 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.613994 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="59783c93-2c17-45c0-accd-18cf0c8962b5" containerName="revoke-keys" Oct 30 22:24:45.617183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.617160 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.669181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.669148 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:24:45.669369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.669184 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:24:45.701819 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.701776 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrlfc\" (UniqueName: \"kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.702013 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.701994 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.702065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.702042 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.718380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.718352 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:24:45.718380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.718358 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:24:45.720096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.720076 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr"] Oct 30 22:24:45.803138 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.803109 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.803283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.803147 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.803283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.803250 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrlfc\" (UniqueName: \"kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.803825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.803801 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.805699 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.805678 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.860441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.860411 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrlfc\" (UniqueName: \"kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc\") pod \"maas-api-revoke-sub-d3fba056-5kjlr\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:45.930173 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:45.930102 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:46.301372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:46.301351 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr"] Oct 30 22:24:46.303721 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:24:46.303693 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bbeb84_ea0b_451b_b2d8_545fab5bc6ba.slice/crio-ef4e9658e2d8e8c8fb57ba0ec2e5f786191d873be37a3679cbccfbe68da75358 WatchSource:0}: Error finding container ef4e9658e2d8e8c8fb57ba0ec2e5f786191d873be37a3679cbccfbe68da75358: Status 404 returned error can't find the container with id ef4e9658e2d8e8c8fb57ba0ec2e5f786191d873be37a3679cbccfbe68da75358 Oct 30 22:24:47.302873 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:47.302839 5508 generic.go:356] "Generic (PLEG): container finished" podID="b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" containerID="4cc7574fc60996e3c6eec0360094deab954789289a4d2097e68612f09a97e982" exitCode=0 Oct 30 22:24:47.303267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:47.302917 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" event={"ID":"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba","Type":"ContainerDied","Data":"4cc7574fc60996e3c6eec0360094deab954789289a4d2097e68612f09a97e982"} Oct 30 22:24:47.303267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:47.302970 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" event={"ID":"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba","Type":"ContainerStarted","Data":"ef4e9658e2d8e8c8fb57ba0ec2e5f786191d873be37a3679cbccfbe68da75358"} Oct 30 22:24:48.483283 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.483250 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:24:48.531580 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.531549 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca\") pod \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " Oct 30 22:24:48.531747 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.531725 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc\" (UniqueName: \"kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc\") pod \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " Oct 30 22:24:48.531831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.531772 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca\") pod \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\" (UID: \"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba\") " Oct 30 22:24:48.532377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.532346 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca" pod "b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" (UID: "b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:48.534016 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.533991 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc" pod "b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" (UID: "b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba"). InnerVolumeSpecName "kube-api-access-nrlfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:48.534282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.534261 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca" pod "b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" (UID: "b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:48.633693 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.633655 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:48.633693 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.633694 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrlfc\" (UniqueName: \"kubernetes.io/projected/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-kube-api-access-nrlfc\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:48.633886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:48.633710 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:49.337043 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:49.337004 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" event={"ID":"b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba","Type":"ContainerDied","Data":"ef4e9658e2d8e8c8fb57ba0ec2e5f786191d873be37a3679cbccfbe68da75358"} Oct 30 22:24:49.337337 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:49.337289 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef4e9658e2d8e8c8fb57ba0ec2e5f786191d873be37a3679cbccfbe68da75358" Oct 30 22:24:49.337536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:24:49.337520 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr" Oct 30 22:25:10.372380 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:25:10.372339 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:25:19.507124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:19.507082 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:19.693720 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:19.693674 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:20.541171 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.541139 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw"] Oct 30 22:25:20.541584 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.541521 5508 cpu_manager.go:414] "containerMap: removing container" podUID="b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" containerName="revoke-keys" Oct 30 22:25:20.541584 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.541532 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" containerName="revoke-keys" Oct 30 22:25:20.541657 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.541619 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" containerName="revoke-keys" Oct 30 22:25:20.544716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.544700 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.584268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.584237 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.584478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.584405 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.584478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.584459 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8s6t\" (UniqueName: \"kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.596606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.596584 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:25:20.596606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.596600 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:20.596768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.596592 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:20.596768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.596631 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:20.647823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.647784 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw"] Oct 30 22:25:20.685740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.685705 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.685949 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.685792 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.685949 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.685817 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8s6t\" (UniqueName: \"kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.686533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.686509 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.688250 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.688228 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.742762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.742731 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8s6t\" (UniqueName: \"kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t\") pod \"maas-api-revoke-sub-ae594979-r9bgw\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:20.857830 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:20.857798 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:21.231912 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:21.231887 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw"] Oct 30 22:25:21.234865 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:25:21.234830 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9290f7ec_691e_49e7_8519_c6e8928d7bb2.slice/crio-37339680531f55e27a9b743aa18a5976d0a5050792988b7027ffda46daaff3e7 WatchSource:0}: Error finding container 37339680531f55e27a9b743aa18a5976d0a5050792988b7027ffda46daaff3e7: Status 404 returned error can't find the container with id 37339680531f55e27a9b743aa18a5976d0a5050792988b7027ffda46daaff3e7 Oct 30 22:25:21.508346 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:21.508294 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" event={"ID":"9290f7ec-691e-49e7-8519-c6e8928d7bb2","Type":"ContainerStarted","Data":"0f21c36afc889c8636288668167f7019b2352d50f76f6629e60e956448c6baca"} Oct 30 22:25:21.508463 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:21.508368 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" event={"ID":"9290f7ec-691e-49e7-8519-c6e8928d7bb2","Type":"ContainerStarted","Data":"37339680531f55e27a9b743aa18a5976d0a5050792988b7027ffda46daaff3e7"} Oct 30 22:25:21.623376 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:21.623288 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" podStartSLOduration=1.623275145 podStartE2EDuration="1.623275145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:21.620425457 +0000 UTC m=+2531.809963162" watchObservedRunningTime="2026-10-30 22:25:21.623275145 +0000 UTC m=+2531.812812858" Oct 30 22:25:22.514829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:22.514792 5508 generic.go:356] "Generic (PLEG): container finished" podID="9290f7ec-691e-49e7-8519-c6e8928d7bb2" containerID="0f21c36afc889c8636288668167f7019b2352d50f76f6629e60e956448c6baca" exitCode=0 Oct 30 22:25:22.515033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:22.514872 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" event={"ID":"9290f7ec-691e-49e7-8519-c6e8928d7bb2","Type":"ContainerDied","Data":"0f21c36afc889c8636288668167f7019b2352d50f76f6629e60e956448c6baca"} Oct 30 22:25:23.659158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.659130 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:23.717881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.717852 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca\") pod \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " Oct 30 22:25:23.718025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.717899 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t\" (UniqueName: \"kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t\") pod \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " Oct 30 22:25:23.718094 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.718026 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca\") pod \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\" (UID: \"9290f7ec-691e-49e7-8519-c6e8928d7bb2\") " Oct 30 22:25:23.718377 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.718355 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca" pod "9290f7ec-691e-49e7-8519-c6e8928d7bb2" (UID: "9290f7ec-691e-49e7-8519-c6e8928d7bb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:23.720035 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.720016 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca" pod "9290f7ec-691e-49e7-8519-c6e8928d7bb2" (UID: "9290f7ec-691e-49e7-8519-c6e8928d7bb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:23.720126 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.720102 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t" pod "9290f7ec-691e-49e7-8519-c6e8928d7bb2" (UID: "9290f7ec-691e-49e7-8519-c6e8928d7bb2"). InnerVolumeSpecName "kube-api-access-k8s6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:23.819628 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.819594 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:23.819780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.819633 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9290f7ec-691e-49e7-8519-c6e8928d7bb2-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:23.819780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:23.819646 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8s6t\" (UniqueName: \"kubernetes.io/projected/9290f7ec-691e-49e7-8519-c6e8928d7bb2-kube-api-access-k8s6t\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:24.525702 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:24.525669 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" Oct 30 22:25:24.525881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:24.525672 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw" event={"ID":"9290f7ec-691e-49e7-8519-c6e8928d7bb2","Type":"ContainerDied","Data":"37339680531f55e27a9b743aa18a5976d0a5050792988b7027ffda46daaff3e7"} Oct 30 22:25:24.525881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:24.525800 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37339680531f55e27a9b743aa18a5976d0a5050792988b7027ffda46daaff3e7" Oct 30 22:25:35.658453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:35.658411 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:40.409727 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:25:40.409657 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:25:40.727397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:40.726849 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:25:40.727397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:40.726956 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:48.611569 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.611533 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf"] Oct 30 22:25:48.612064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.611916 5508 cpu_manager.go:414] "containerMap: removing container" podUID="9290f7ec-691e-49e7-8519-c6e8928d7bb2" containerName="revoke-keys" Oct 30 22:25:48.612064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.611927 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9290f7ec-691e-49e7-8519-c6e8928d7bb2" containerName="revoke-keys" Oct 30 22:25:48.612064 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.612017 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9290f7ec-691e-49e7-8519-c6e8928d7bb2" containerName="revoke-keys" Oct 30 22:25:48.615190 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.615169 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.670373 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.670328 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:25:48.718563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.718533 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.718716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.718598 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75vfh\" (UniqueName: \"kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.718716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.718694 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.719552 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.719534 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:25:48.719647 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.719622 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:25:48.719734 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.719719 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:25:48.721448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.721428 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf"] Oct 30 22:25:48.819990 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.819966 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75vfh\" (UniqueName: \"kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.820116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.820001 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.820116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.820055 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.820697 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.820677 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.822633 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.822608 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.876709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.876658 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75vfh\" (UniqueName: \"kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh\") pod \"maas-api-revoke-sub-a4aef7d7-66cdf\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:48.928355 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:48.928335 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:49.301146 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:49.301120 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf"] Oct 30 22:25:49.303531 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:25:49.303502 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1651be40_be6c_498b_9bcb_7649a8cecae3.slice/crio-ce324ce2407ba30133a412193e8a7d652dd1443cad74e9314d74777606bff0c1 WatchSource:0}: Error finding container ce324ce2407ba30133a412193e8a7d652dd1443cad74e9314d74777606bff0c1: Status 404 returned error can't find the container with id ce324ce2407ba30133a412193e8a7d652dd1443cad74e9314d74777606bff0c1 Oct 30 22:25:49.683702 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:49.683672 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" event={"ID":"1651be40-be6c-498b-9bcb-7649a8cecae3","Type":"ContainerStarted","Data":"dce66e7741b581e5f8bb71c67fde1b0765181b60433c124ab1966477dd46d41a"} Oct 30 22:25:49.684069 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:49.683722 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" event={"ID":"1651be40-be6c-498b-9bcb-7649a8cecae3","Type":"ContainerStarted","Data":"ce324ce2407ba30133a412193e8a7d652dd1443cad74e9314d74777606bff0c1"} Oct 30 22:25:49.798342 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:49.798268 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" podStartSLOduration=1.798253968 podStartE2EDuration="1.798253968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:49.79665346 +0000 UTC m=+2559.986191145" watchObservedRunningTime="2026-10-30 22:25:49.798253968 +0000 UTC m=+2559.987791677" Oct 30 22:25:50.690462 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:50.690431 5508 generic.go:356] "Generic (PLEG): container finished" podID="1651be40-be6c-498b-9bcb-7649a8cecae3" containerID="dce66e7741b581e5f8bb71c67fde1b0765181b60433c124ab1966477dd46d41a" exitCode=0 Oct 30 22:25:50.690864 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:50.690519 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" event={"ID":"1651be40-be6c-498b-9bcb-7649a8cecae3","Type":"ContainerDied","Data":"dce66e7741b581e5f8bb71c67fde1b0765181b60433c124ab1966477dd46d41a"} Oct 30 22:25:51.828471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.828446 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:51.949149 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.949077 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh\" (UniqueName: \"kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh\") pod \"1651be40-be6c-498b-9bcb-7649a8cecae3\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " Oct 30 22:25:51.949330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.949158 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca\") pod \"1651be40-be6c-498b-9bcb-7649a8cecae3\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " Oct 30 22:25:51.949330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.949213 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca\") pod \"1651be40-be6c-498b-9bcb-7649a8cecae3\" (UID: \"1651be40-be6c-498b-9bcb-7649a8cecae3\") " Oct 30 22:25:51.949795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.949632 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca" pod "1651be40-be6c-498b-9bcb-7649a8cecae3" (UID: "1651be40-be6c-498b-9bcb-7649a8cecae3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:51.951796 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.951754 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh" pod "1651be40-be6c-498b-9bcb-7649a8cecae3" (UID: "1651be40-be6c-498b-9bcb-7649a8cecae3"). InnerVolumeSpecName "kube-api-access-75vfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:51.951964 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:51.951933 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca" pod "1651be40-be6c-498b-9bcb-7649a8cecae3" (UID: "1651be40-be6c-498b-9bcb-7649a8cecae3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:52.050433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:52.050402 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75vfh\" (UniqueName: \"kubernetes.io/projected/1651be40-be6c-498b-9bcb-7649a8cecae3-kube-api-access-75vfh\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.050433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:52.050429 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.050654 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:52.050441 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1651be40-be6c-498b-9bcb-7649a8cecae3-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:52.703115 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:52.703088 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" Oct 30 22:25:52.703115 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:52.703098 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf" event={"ID":"1651be40-be6c-498b-9bcb-7649a8cecae3","Type":"ContainerDied","Data":"ce324ce2407ba30133a412193e8a7d652dd1443cad74e9314d74777606bff0c1"} Oct 30 22:25:52.703344 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:25:52.703143 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce324ce2407ba30133a412193e8a7d652dd1443cad74e9314d74777606bff0c1" Oct 30 22:26:10.403621 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:26:10.403535 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:26:10.726529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:10.726401 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:26:10.726868 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:10.726817 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:19.199591 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.199553 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85"] Oct 30 22:26:19.201958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.200014 5508 cpu_manager.go:414] "containerMap: removing container" podUID="1651be40-be6c-498b-9bcb-7649a8cecae3" containerName="revoke-keys" Oct 30 22:26:19.201958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.200032 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1651be40-be6c-498b-9bcb-7649a8cecae3" containerName="revoke-keys" Oct 30 22:26:19.201958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.200184 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1651be40-be6c-498b-9bcb-7649a8cecae3" containerName="revoke-keys" Oct 30 22:26:19.203325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.203294 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.234105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.234075 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.234203 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.234120 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fc9\" (UniqueName: \"kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.234360 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.234332 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.256209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.256190 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:19.305533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.305498 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:19.305533 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.305515 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:19.305869 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.305842 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:26:19.307507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.307485 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85"] Oct 30 22:26:19.335022 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.334996 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.335125 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.335037 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.335125 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.335057 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fc9\" (UniqueName: \"kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.335776 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.335697 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.337249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.337229 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.391169 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.391144 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fc9\" (UniqueName: \"kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9\") pod \"maas-api-revoke-sub-79805ca9-v6h85\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.516182 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.516095 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:19.891012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:19.890976 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85"] Oct 30 22:26:19.894357 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:26:19.894300 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7f74f6_7cac_440c_8883_2e1f1c26c7fa.slice/crio-4861c6b6e9add55bd0562b9046383466f134180dfed6da5d18b061c80bdf3c32 WatchSource:0}: Error finding container 4861c6b6e9add55bd0562b9046383466f134180dfed6da5d18b061c80bdf3c32: Status 404 returned error can't find the container with id 4861c6b6e9add55bd0562b9046383466f134180dfed6da5d18b061c80bdf3c32 Oct 30 22:26:20.862823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:20.862789 5508 generic.go:356] "Generic (PLEG): container finished" podID="cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" containerID="727075d02f542cca44b724e220a40bac0094061e80daa3bd1a9beb4a57f49f57" exitCode=0 Oct 30 22:26:20.863185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:20.862879 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" event={"ID":"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa","Type":"ContainerDied","Data":"727075d02f542cca44b724e220a40bac0094061e80daa3bd1a9beb4a57f49f57"} Oct 30 22:26:20.863185 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:20.862933 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" event={"ID":"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa","Type":"ContainerStarted","Data":"4861c6b6e9add55bd0562b9046383466f134180dfed6da5d18b061c80bdf3c32"} Oct 30 22:26:22.015947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.015921 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:22.061671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.061643 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca\") pod \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " Oct 30 22:26:22.061815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.061703 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca\") pod \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " Oct 30 22:26:22.061815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.061766 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9\" (UniqueName: \"kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9\") pod \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\" (UID: \"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa\") " Oct 30 22:26:22.062007 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.061984 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca" pod "cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" (UID: "cc7f74f6-7cac-440c-8883-2e1f1c26c7fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:22.062103 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.062084 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:22.064057 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.064030 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca" pod "cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" (UID: "cc7f74f6-7cac-440c-8883-2e1f1c26c7fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:22.064160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.064059 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9" pod "cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" (UID: "cc7f74f6-7cac-440c-8883-2e1f1c26c7fa"). InnerVolumeSpecName "kube-api-access-97fc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:22.163142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.163061 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:22.163142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.163102 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97fc9\" (UniqueName: \"kubernetes.io/projected/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa-kube-api-access-97fc9\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:22.873808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.873780 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" Oct 30 22:26:22.873808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.873784 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85" event={"ID":"cc7f74f6-7cac-440c-8883-2e1f1c26c7fa","Type":"ContainerDied","Data":"4861c6b6e9add55bd0562b9046383466f134180dfed6da5d18b061c80bdf3c32"} Oct 30 22:26:22.874016 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:22.873833 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4861c6b6e9add55bd0562b9046383466f134180dfed6da5d18b061c80bdf3c32" Oct 30 22:26:34.246024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:34.245969 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:36.047079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:36.047024 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:39.901623 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:39.901591 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2"] Oct 30 22:26:39.902065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:39.901982 5508 cpu_manager.go:414] "containerMap: removing container" podUID="cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" containerName="revoke-keys" Oct 30 22:26:39.902065 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:39.901992 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" containerName="revoke-keys" Oct 30 22:26:39.902186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:39.902067 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" containerName="revoke-keys" Oct 30 22:26:39.906332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:39.906282 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:39.961896 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:39.961553 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:40.010425 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.010392 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:40.010587 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.010464 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:40.010629 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.010614 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:26:40.012430 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.012410 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2"] Oct 30 22:26:40.075206 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.075174 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.075376 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.075225 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fdh\" (UniqueName: \"kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.075490 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.075474 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.176201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.176129 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.176201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.176162 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.176201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.176191 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fdh\" (UniqueName: \"kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.176933 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.176898 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.178663 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.178642 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.232979 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.232946 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fdh\" (UniqueName: \"kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh\") pod \"maas-api-revoke-sub-0b7e8edb-mx4s2\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.372263 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:26:40.372229 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:26:40.519129 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.519058 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:40.891161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.891135 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2"] Oct 30 22:26:40.892973 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:26:40.892949 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc16e57_2e09_4cdc_8a88_4f6b6fd8514d.slice/crio-e77b443124538e86314e5414c5dc126bba449b1e7dd5e902688d2702e4ffbd02 WatchSource:0}: Error finding container e77b443124538e86314e5414c5dc126bba449b1e7dd5e902688d2702e4ffbd02: Status 404 returned error can't find the container with id e77b443124538e86314e5414c5dc126bba449b1e7dd5e902688d2702e4ffbd02 Oct 30 22:26:40.971817 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:40.971792 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" event={"ID":"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d","Type":"ContainerStarted","Data":"e77b443124538e86314e5414c5dc126bba449b1e7dd5e902688d2702e4ffbd02"} Oct 30 22:26:41.977846 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:41.977810 5508 generic.go:356] "Generic (PLEG): container finished" podID="4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" containerID="40a66a31a2676d1952b08f77358fcfde8b2831dd3d38ed5d91279aa07f9f60f3" exitCode=0 Oct 30 22:26:41.978232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:41.977853 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" event={"ID":"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d","Type":"ContainerDied","Data":"40a66a31a2676d1952b08f77358fcfde8b2831dd3d38ed5d91279aa07f9f60f3"} Oct 30 22:26:43.118219 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.118196 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:43.204601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.204577 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca\") pod \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " Oct 30 22:26:43.204723 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.204641 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh\" (UniqueName: \"kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh\") pod \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " Oct 30 22:26:43.204723 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.204714 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca\") pod \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\" (UID: \"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d\") " Oct 30 22:26:43.204905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.204883 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca" pod "4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" (UID: "4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:43.206750 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.206720 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh" pod "4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" (UID: "4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d"). InnerVolumeSpecName "kube-api-access-g5fdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:43.206871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.206783 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca" pod "4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" (UID: "4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:43.305529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.305432 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:43.305529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.305467 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5fdh\" (UniqueName: \"kubernetes.io/projected/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-kube-api-access-g5fdh\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:43.305529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.305482 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:43.988253 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.988225 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" Oct 30 22:26:43.988409 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.988263 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2" event={"ID":"4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d","Type":"ContainerDied","Data":"e77b443124538e86314e5414c5dc126bba449b1e7dd5e902688d2702e4ffbd02"} Oct 30 22:26:43.988409 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:43.988325 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e77b443124538e86314e5414c5dc126bba449b1e7dd5e902688d2702e4ffbd02" Oct 30 22:26:51.272601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.272516 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr"] Oct 30 22:26:51.272974 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.272918 5508 cpu_manager.go:414] "containerMap: removing container" podUID="4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" containerName="revoke-keys" Oct 30 22:26:51.272974 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.272929 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" containerName="revoke-keys" Oct 30 22:26:51.273046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.272988 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" containerName="revoke-keys" Oct 30 22:26:51.276170 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.276154 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.380017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.379986 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:26:51.380017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.380004 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:26:51.380225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.380039 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:26:51.380225 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.380048 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:26:51.381949 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.381929 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr"] Oct 30 22:26:51.383014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.382997 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bnq\" (UniqueName: \"kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.383066 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.383031 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.383122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.383110 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.484059 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.484033 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bnq\" (UniqueName: \"kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.484245 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.484078 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.484245 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.484129 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.484849 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.484827 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.486604 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.486583 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.541162 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.541101 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bnq\" (UniqueName: \"kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq\") pod \"maas-api-revoke-sub-3d4156f1-f8gsr\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.589387 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.589361 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:51.960722 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:51.960693 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr"] Oct 30 22:26:51.963969 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:26:51.963931 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438e6894_1466_4b57_8cd4_54227c7ae1a2.slice/crio-4a89af15c44535262041d533cdf6ee0c851ead1a45172bde3534f110775f2571 WatchSource:0}: Error finding container 4a89af15c44535262041d533cdf6ee0c851ead1a45172bde3534f110775f2571: Status 404 returned error can't find the container with id 4a89af15c44535262041d533cdf6ee0c851ead1a45172bde3534f110775f2571 Oct 30 22:26:52.029247 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:52.029214 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" event={"ID":"438e6894-1466-4b57-8cd4-54227c7ae1a2","Type":"ContainerStarted","Data":"4a89af15c44535262041d533cdf6ee0c851ead1a45172bde3534f110775f2571"} Oct 30 22:26:53.035507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:53.035468 5508 generic.go:356] "Generic (PLEG): container finished" podID="438e6894-1466-4b57-8cd4-54227c7ae1a2" containerID="ac63862b273465a2d380c3a483c4ed77aaaeee7a065789fabc6111a4cd91bdad" exitCode=0 Oct 30 22:26:53.035851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:53.035551 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" event={"ID":"438e6894-1466-4b57-8cd4-54227c7ae1a2","Type":"ContainerDied","Data":"ac63862b273465a2d380c3a483c4ed77aaaeee7a065789fabc6111a4cd91bdad"} Oct 30 22:26:54.181245 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.181212 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:54.310142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.310057 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq\" (UniqueName: \"kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq\") pod \"438e6894-1466-4b57-8cd4-54227c7ae1a2\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " Oct 30 22:26:54.310142 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.310111 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca\") pod \"438e6894-1466-4b57-8cd4-54227c7ae1a2\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " Oct 30 22:26:54.310413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.310148 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca\") pod \"438e6894-1466-4b57-8cd4-54227c7ae1a2\" (UID: \"438e6894-1466-4b57-8cd4-54227c7ae1a2\") " Oct 30 22:26:54.310590 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.310568 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca" pod "438e6894-1466-4b57-8cd4-54227c7ae1a2" (UID: "438e6894-1466-4b57-8cd4-54227c7ae1a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:54.312279 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.312249 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq" pod "438e6894-1466-4b57-8cd4-54227c7ae1a2" (UID: "438e6894-1466-4b57-8cd4-54227c7ae1a2"). InnerVolumeSpecName "kube-api-access-k2bnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:54.312542 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.312521 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca" pod "438e6894-1466-4b57-8cd4-54227c7ae1a2" (UID: "438e6894-1466-4b57-8cd4-54227c7ae1a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:54.411456 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.411432 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bnq\" (UniqueName: \"kubernetes.io/projected/438e6894-1466-4b57-8cd4-54227c7ae1a2-kube-api-access-k2bnq\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:54.411456 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.411456 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:54.411456 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:54.411466 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/438e6894-1466-4b57-8cd4-54227c7ae1a2-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:55.046333 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:55.046269 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" Oct 30 22:26:55.046521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:55.046271 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr" event={"ID":"438e6894-1466-4b57-8cd4-54227c7ae1a2","Type":"ContainerDied","Data":"4a89af15c44535262041d533cdf6ee0c851ead1a45172bde3534f110775f2571"} Oct 30 22:26:55.046521 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:26:55.046471 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a89af15c44535262041d533cdf6ee0c851ead1a45172bde3534f110775f2571" Oct 30 22:27:08.263432 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:08.263398 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:08.263913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:08.263457 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:10.047497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:10.047461 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c"] Oct 30 22:27:10.096940 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:10.096910 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-c7l5c"] Oct 30 22:27:10.372044 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:27:10.372009 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:27:11.951372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:11.951340 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5607ad-bd58-439a-9e5c-e70ff5608a04" path="/var/lib/kubelet/pods/ac5607ad-bd58-439a-9e5c-e70ff5608a04/volumes" Oct 30 22:27:12.418576 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.418538 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr"] Oct 30 22:27:12.467921 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.467883 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-f8gsr"] Oct 30 22:27:12.541223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.541191 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj"] Oct 30 22:27:12.541636 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.541622 5508 cpu_manager.go:414] "containerMap: removing container" podUID="438e6894-1466-4b57-8cd4-54227c7ae1a2" containerName="revoke-keys" Oct 30 22:27:12.541681 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.541641 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438e6894-1466-4b57-8cd4-54227c7ae1a2" containerName="revoke-keys" Oct 30 22:27:12.541737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.541727 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="438e6894-1466-4b57-8cd4-54227c7ae1a2" containerName="revoke-keys" Oct 30 22:27:12.544983 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.544968 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.597496 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.597454 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:12.597662 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.597589 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:12.597740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.597654 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:12.597796 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.597739 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:27:12.600499 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.600470 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.600641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.600613 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2jj\" (UniqueName: \"kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.600774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.600657 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.648489 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.648458 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj"] Oct 30 22:27:12.702143 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.702084 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.702259 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.702143 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2jj\" (UniqueName: \"kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.702259 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.702172 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.702771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.702743 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.704756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.704734 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.760460 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.760431 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2jj\" (UniqueName: \"kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj\") pod \"maas-api-revoke-sub-3d4156f1-rkftj\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:12.857739 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:12.857712 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:13.233918 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:13.233761 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj"] Oct 30 22:27:13.236500 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:27:13.236464 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b205cd_8509_452f_bd1c_8a50bd261923.slice/crio-6dea55e334df9363c2af5d85a1ba171f38d6d6af1f28a077a494e296ad5719e2 WatchSource:0}: Error finding container 6dea55e334df9363c2af5d85a1ba171f38d6d6af1f28a077a494e296ad5719e2: Status 404 returned error can't find the container with id 6dea55e334df9363c2af5d85a1ba171f38d6d6af1f28a077a494e296ad5719e2 Oct 30 22:27:13.953516 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:13.953486 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438e6894-1466-4b57-8cd4-54227c7ae1a2" path="/var/lib/kubelet/pods/438e6894-1466-4b57-8cd4-54227c7ae1a2/volumes" Oct 30 22:27:14.153557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:14.153523 5508 generic.go:356] "Generic (PLEG): container finished" podID="c0b205cd-8509-452f-bd1c-8a50bd261923" containerID="410ab08aa7ab121637ad0eefc8dbe76295d190bd8f41c90f7acc3c130510756d" exitCode=0 Oct 30 22:27:14.153729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:14.153613 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" event={"ID":"c0b205cd-8509-452f-bd1c-8a50bd261923","Type":"ContainerDied","Data":"410ab08aa7ab121637ad0eefc8dbe76295d190bd8f41c90f7acc3c130510756d"} Oct 30 22:27:14.153729 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:14.153667 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" event={"ID":"c0b205cd-8509-452f-bd1c-8a50bd261923","Type":"ContainerStarted","Data":"6dea55e334df9363c2af5d85a1ba171f38d6d6af1f28a077a494e296ad5719e2"} Oct 30 22:27:15.296363 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.296326 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:15.430146 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.430079 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca\") pod \"c0b205cd-8509-452f-bd1c-8a50bd261923\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " Oct 30 22:27:15.430285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.430162 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca\") pod \"c0b205cd-8509-452f-bd1c-8a50bd261923\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " Oct 30 22:27:15.430285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.430222 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj\" (UniqueName: \"kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj\") pod \"c0b205cd-8509-452f-bd1c-8a50bd261923\" (UID: \"c0b205cd-8509-452f-bd1c-8a50bd261923\") " Oct 30 22:27:15.430550 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.430527 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca" pod "c0b205cd-8509-452f-bd1c-8a50bd261923" (UID: "c0b205cd-8509-452f-bd1c-8a50bd261923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:15.432412 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.432389 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca" pod "c0b205cd-8509-452f-bd1c-8a50bd261923" (UID: "c0b205cd-8509-452f-bd1c-8a50bd261923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:15.432545 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.432529 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj" pod "c0b205cd-8509-452f-bd1c-8a50bd261923" (UID: "c0b205cd-8509-452f-bd1c-8a50bd261923"). InnerVolumeSpecName "kube-api-access-jd2jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:15.531487 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.531460 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.531621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.531489 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2jj\" (UniqueName: \"kubernetes.io/projected/c0b205cd-8509-452f-bd1c-8a50bd261923-kube-api-access-jd2jj\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:15.531621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:15.531505 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b205cd-8509-452f-bd1c-8a50bd261923-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:16.165228 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:16.165203 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" Oct 30 22:27:16.165399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:16.165239 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj" event={"ID":"c0b205cd-8509-452f-bd1c-8a50bd261923","Type":"ContainerDied","Data":"6dea55e334df9363c2af5d85a1ba171f38d6d6af1f28a077a494e296ad5719e2"} Oct 30 22:27:16.165399 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:16.165283 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dea55e334df9363c2af5d85a1ba171f38d6d6af1f28a077a494e296ad5719e2" Oct 30 22:27:36.411618 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:36.411582 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:40.372694 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:27:40.372659 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:27:50.358472 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.358435 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt"] Oct 30 22:27:50.359019 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.358854 5508 cpu_manager.go:414] "containerMap: removing container" podUID="c0b205cd-8509-452f-bd1c-8a50bd261923" containerName="revoke-keys" Oct 30 22:27:50.359019 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.358866 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b205cd-8509-452f-bd1c-8a50bd261923" containerName="revoke-keys" Oct 30 22:27:50.359019 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.358956 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b205cd-8509-452f-bd1c-8a50bd261923" containerName="revoke-keys" Oct 30 22:27:50.362322 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.362282 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.374655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.374628 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt"] Oct 30 22:27:50.414189 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.414163 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:27:50.414350 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.414277 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:27:50.414350 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.414340 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:27:50.414440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.414340 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:27:50.417520 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.417496 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.417618 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.417552 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llsr5\" (UniqueName: \"kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.417666 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.417615 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.518851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.518823 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.518995 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.518875 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llsr5\" (UniqueName: \"kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.518995 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.518906 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.519516 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.519484 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.521275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.521255 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.576098 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.576068 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llsr5\" (UniqueName: \"kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5\") pod \"maas-api-revoke-sub-e2fc9c5b-cqrqt\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:50.676400 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:50.676336 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:51.052293 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:51.052267 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt"] Oct 30 22:27:51.054169 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:27:51.054138 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode116f826_e5e9_4f7c_8274_0c7214e38972.slice/crio-9323e572c42c38e36f2920e0319b5b77b31035b305e3e9ff4acb82b8ca512a53 WatchSource:0}: Error finding container 9323e572c42c38e36f2920e0319b5b77b31035b305e3e9ff4acb82b8ca512a53: Status 404 returned error can't find the container with id 9323e572c42c38e36f2920e0319b5b77b31035b305e3e9ff4acb82b8ca512a53 Oct 30 22:27:51.362077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:51.362040 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" event={"ID":"e116f826-e5e9-4f7c-8274-0c7214e38972","Type":"ContainerStarted","Data":"9323e572c42c38e36f2920e0319b5b77b31035b305e3e9ff4acb82b8ca512a53"} Oct 30 22:27:52.369044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:52.369012 5508 generic.go:356] "Generic (PLEG): container finished" podID="e116f826-e5e9-4f7c-8274-0c7214e38972" containerID="043458562cfbe7b3bbb74032015a83737fcd6a74ed5cea6771162b10577b30f2" exitCode=0 Oct 30 22:27:52.369479 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:52.369104 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" event={"ID":"e116f826-e5e9-4f7c-8274-0c7214e38972","Type":"ContainerDied","Data":"043458562cfbe7b3bbb74032015a83737fcd6a74ed5cea6771162b10577b30f2"} Oct 30 22:27:53.514890 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.514866 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:53.546215 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.546186 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5\" (UniqueName: \"kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5\") pod \"e116f826-e5e9-4f7c-8274-0c7214e38972\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " Oct 30 22:27:53.546395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.546241 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca\") pod \"e116f826-e5e9-4f7c-8274-0c7214e38972\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " Oct 30 22:27:53.546395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.546287 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca\") pod \"e116f826-e5e9-4f7c-8274-0c7214e38972\" (UID: \"e116f826-e5e9-4f7c-8274-0c7214e38972\") " Oct 30 22:27:53.546684 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.546655 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca" pod "e116f826-e5e9-4f7c-8274-0c7214e38972" (UID: "e116f826-e5e9-4f7c-8274-0c7214e38972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:27:53.548433 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.548412 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5" pod "e116f826-e5e9-4f7c-8274-0c7214e38972" (UID: "e116f826-e5e9-4f7c-8274-0c7214e38972"). InnerVolumeSpecName "kube-api-access-llsr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:27:53.548551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.548532 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca" pod "e116f826-e5e9-4f7c-8274-0c7214e38972" (UID: "e116f826-e5e9-4f7c-8274-0c7214e38972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:27:53.647881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.647805 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:53.647881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.647836 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llsr5\" (UniqueName: \"kubernetes.io/projected/e116f826-e5e9-4f7c-8274-0c7214e38972-kube-api-access-llsr5\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:53.647881 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:53.647852 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e116f826-e5e9-4f7c-8274-0c7214e38972-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:27:54.379916 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:54.379886 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" Oct 30 22:27:54.379916 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:54.379898 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt" event={"ID":"e116f826-e5e9-4f7c-8274-0c7214e38972","Type":"ContainerDied","Data":"9323e572c42c38e36f2920e0319b5b77b31035b305e3e9ff4acb82b8ca512a53"} Oct 30 22:27:54.380126 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:27:54.379965 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9323e572c42c38e36f2920e0319b5b77b31035b305e3e9ff4acb82b8ca512a53" Oct 30 22:28:01.088659 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.088617 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl"] Oct 30 22:28:01.089072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.089059 5508 cpu_manager.go:414] "containerMap: removing container" podUID="e116f826-e5e9-4f7c-8274-0c7214e38972" containerName="revoke-keys" Oct 30 22:28:01.089122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.089074 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e116f826-e5e9-4f7c-8274-0c7214e38972" containerName="revoke-keys" Oct 30 22:28:01.089160 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.089148 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e116f826-e5e9-4f7c-8274-0c7214e38972" containerName="revoke-keys" Oct 30 22:28:01.092483 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.092466 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.119859 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.119828 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4c5\" (UniqueName: \"kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.120005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.119864 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.120005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.119937 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.144149 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.144123 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:01.144448 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.144428 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:28:01.144528 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.144428 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:28:01.144528 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.144458 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:28:01.195428 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.195403 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl"] Oct 30 22:28:01.221004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.220972 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.221114 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.221074 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4c5\" (UniqueName: \"kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.221114 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.221092 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.221749 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.221727 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.223390 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.223368 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.278183 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.278160 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4c5\" (UniqueName: \"kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5\") pod \"maas-api-revoke-sub-5a29efb3-4qldl\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.406217 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.406135 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:01.781954 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:01.781924 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl"] Oct 30 22:28:01.784740 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:28:01.784714 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ecdfb2_087e_4113_ad2e_015bf4da26ef.slice/crio-b7a15936c07c4db5bd384d1aeb6c5b73d4eea3060bfabbf70343fa4c5c45af3c WatchSource:0}: Error finding container b7a15936c07c4db5bd384d1aeb6c5b73d4eea3060bfabbf70343fa4c5c45af3c: Status 404 returned error can't find the container with id b7a15936c07c4db5bd384d1aeb6c5b73d4eea3060bfabbf70343fa4c5c45af3c Oct 30 22:28:02.421921 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:02.421825 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" event={"ID":"48ecdfb2-087e-4113-ad2e-015bf4da26ef","Type":"ContainerStarted","Data":"5832a97d8b2bc7a633e981823336d925827c84f8f143bda93946c50eb3a047c8"} Oct 30 22:28:02.421921 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:02.421891 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" event={"ID":"48ecdfb2-087e-4113-ad2e-015bf4da26ef","Type":"ContainerStarted","Data":"b7a15936c07c4db5bd384d1aeb6c5b73d4eea3060bfabbf70343fa4c5c45af3c"} Oct 30 22:28:02.545103 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:02.545048 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" podStartSLOduration=1.54503342 podStartE2EDuration="1.54503342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:02.543398805 +0000 UTC m=+2692.732936498" watchObservedRunningTime="2026-10-30 22:28:02.54503342 +0000 UTC m=+2692.734571096" Oct 30 22:28:03.429903 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:03.429868 5508 generic.go:356] "Generic (PLEG): container finished" podID="48ecdfb2-087e-4113-ad2e-015bf4da26ef" containerID="5832a97d8b2bc7a633e981823336d925827c84f8f143bda93946c50eb3a047c8" exitCode=0 Oct 30 22:28:03.430410 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:03.429942 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" event={"ID":"48ecdfb2-087e-4113-ad2e-015bf4da26ef","Type":"ContainerDied","Data":"5832a97d8b2bc7a633e981823336d925827c84f8f143bda93946c50eb3a047c8"} Oct 30 22:28:04.848617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.848255 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:04.862922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.862900 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca\") pod \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " Oct 30 22:28:04.863113 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.863096 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca\") pod \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " Oct 30 22:28:04.863274 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.863257 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5\" (UniqueName: \"kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5\") pod \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\" (UID: \"48ecdfb2-087e-4113-ad2e-015bf4da26ef\") " Oct 30 22:28:04.863592 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.863569 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca" pod "48ecdfb2-087e-4113-ad2e-015bf4da26ef" (UID: "48ecdfb2-087e-4113-ad2e-015bf4da26ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:04.863897 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.863881 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.865246 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.865226 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca" pod "48ecdfb2-087e-4113-ad2e-015bf4da26ef" (UID: "48ecdfb2-087e-4113-ad2e-015bf4da26ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:04.865601 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.865582 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5" pod "48ecdfb2-087e-4113-ad2e-015bf4da26ef" (UID: "48ecdfb2-087e-4113-ad2e-015bf4da26ef"). InnerVolumeSpecName "kube-api-access-bx4c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:04.964762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.964708 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ecdfb2-087e-4113-ad2e-015bf4da26ef-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.964762 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:04.964738 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4c5\" (UniqueName: \"kubernetes.io/projected/48ecdfb2-087e-4113-ad2e-015bf4da26ef-kube-api-access-bx4c5\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:05.456561 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:05.456529 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" Oct 30 22:28:05.456561 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:05.456549 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl" event={"ID":"48ecdfb2-087e-4113-ad2e-015bf4da26ef","Type":"ContainerDied","Data":"b7a15936c07c4db5bd384d1aeb6c5b73d4eea3060bfabbf70343fa4c5c45af3c"} Oct 30 22:28:05.456774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:05.456600 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7a15936c07c4db5bd384d1aeb6c5b73d4eea3060bfabbf70343fa4c5c45af3c" Oct 30 22:28:10.373111 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:28:10.373067 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:28:10.979280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:10.979253 5508 scope.go:122] "RemoveContainer" containerID="0664525dad338afa5de3e2da248b3ce8590c78134aa6c4fbffa716e6149c4b0a" Oct 30 22:28:18.033774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.033733 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj"] Oct 30 22:28:18.082668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.082629 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rkftj"] Oct 30 22:28:18.149705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.149669 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk"] Oct 30 22:28:18.150161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.150145 5508 cpu_manager.go:414] "containerMap: removing container" podUID="48ecdfb2-087e-4113-ad2e-015bf4da26ef" containerName="revoke-keys" Oct 30 22:28:18.150207 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.150163 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ecdfb2-087e-4113-ad2e-015bf4da26ef" containerName="revoke-keys" Oct 30 22:28:18.150247 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.150238 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ecdfb2-087e-4113-ad2e-015bf4da26ef" containerName="revoke-keys" Oct 30 22:28:18.154777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.154758 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.208419 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.208394 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:28:18.208585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.208430 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:28:18.208667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.208510 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:28:18.208667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.208577 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:28:18.210733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.210704 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk"] Oct 30 22:28:18.298104 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.298005 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.298104 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.298069 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4j7\" (UniqueName: \"kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.298330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.298171 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.399076 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.399044 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.399260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.399104 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.399260 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.399149 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4j7\" (UniqueName: \"kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.399814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.399789 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.401618 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.401600 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.456173 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.456143 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4j7\" (UniqueName: \"kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7\") pod \"maas-api-revoke-sub-3d4156f1-nqjbk\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.467976 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.467955 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:18.878778 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:18.878752 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk"] Oct 30 22:28:18.881144 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:28:18.881117 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb106d7cc_0c83_4238_814f_ee70366c7620.slice/crio-03b3860e106abb1347eaf4dc87cbcf00f9e57aed6acffc5a1e70c4c8fb59e8a5 WatchSource:0}: Error finding container 03b3860e106abb1347eaf4dc87cbcf00f9e57aed6acffc5a1e70c4c8fb59e8a5: Status 404 returned error can't find the container with id 03b3860e106abb1347eaf4dc87cbcf00f9e57aed6acffc5a1e70c4c8fb59e8a5 Oct 30 22:28:19.049980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.049950 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf"] Oct 30 22:28:19.056943 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.056915 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-slkhf"] Oct 30 22:28:19.533872 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.533770 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" event={"ID":"b106d7cc-0c83-4238-814f-ee70366c7620","Type":"ContainerStarted","Data":"ff0cd47620ed4867710b2f933b3fae4b1883928d586022e2eca608f35dfdf691"} Oct 30 22:28:19.533872 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.533857 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" event={"ID":"b106d7cc-0c83-4238-814f-ee70366c7620","Type":"ContainerStarted","Data":"03b3860e106abb1347eaf4dc87cbcf00f9e57aed6acffc5a1e70c4c8fb59e8a5"} Oct 30 22:28:19.650272 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.650218 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" podStartSLOduration=1.650205066 podStartE2EDuration="1.650205066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:19.650099445 +0000 UTC m=+2709.839637190" watchObservedRunningTime="2026-10-30 22:28:19.650205066 +0000 UTC m=+2709.839742735" Oct 30 22:28:19.952454 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.952422 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f006a18-c91f-49e6-9759-54b8aaf008f2" path="/var/lib/kubelet/pods/7f006a18-c91f-49e6-9759-54b8aaf008f2/volumes" Oct 30 22:28:19.952828 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:19.952814 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b205cd-8509-452f-bd1c-8a50bd261923" path="/var/lib/kubelet/pods/c0b205cd-8509-452f-bd1c-8a50bd261923/volumes" Oct 30 22:28:20.541008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:20.540971 5508 generic.go:356] "Generic (PLEG): container finished" podID="b106d7cc-0c83-4238-814f-ee70366c7620" containerID="ff0cd47620ed4867710b2f933b3fae4b1883928d586022e2eca608f35dfdf691" exitCode=0 Oct 30 22:28:20.541425 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:20.541028 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" event={"ID":"b106d7cc-0c83-4238-814f-ee70366c7620","Type":"ContainerDied","Data":"ff0cd47620ed4867710b2f933b3fae4b1883928d586022e2eca608f35dfdf691"} Oct 30 22:28:21.680214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.680192 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:21.831519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.831491 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7\" (UniqueName: \"kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7\") pod \"b106d7cc-0c83-4238-814f-ee70366c7620\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " Oct 30 22:28:21.831672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.831564 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca\") pod \"b106d7cc-0c83-4238-814f-ee70366c7620\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " Oct 30 22:28:21.831672 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.831609 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca\") pod \"b106d7cc-0c83-4238-814f-ee70366c7620\" (UID: \"b106d7cc-0c83-4238-814f-ee70366c7620\") " Oct 30 22:28:21.831970 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.831942 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca" pod "b106d7cc-0c83-4238-814f-ee70366c7620" (UID: "b106d7cc-0c83-4238-814f-ee70366c7620"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:21.833746 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.833720 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7" pod "b106d7cc-0c83-4238-814f-ee70366c7620" (UID: "b106d7cc-0c83-4238-814f-ee70366c7620"). InnerVolumeSpecName "kube-api-access-fp4j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:21.833841 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.833830 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca" pod "b106d7cc-0c83-4238-814f-ee70366c7620" (UID: "b106d7cc-0c83-4238-814f-ee70366c7620"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:21.933251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.933210 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:21.933251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.933242 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b106d7cc-0c83-4238-814f-ee70366c7620-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:21.933251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:21.933251 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp4j7\" (UniqueName: \"kubernetes.io/projected/b106d7cc-0c83-4238-814f-ee70366c7620-kube-api-access-fp4j7\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:22.552680 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:22.552641 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" event={"ID":"b106d7cc-0c83-4238-814f-ee70366c7620","Type":"ContainerDied","Data":"03b3860e106abb1347eaf4dc87cbcf00f9e57aed6acffc5a1e70c4c8fb59e8a5"} Oct 30 22:28:22.552680 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:22.552681 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk" Oct 30 22:28:22.552947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:22.552693 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b3860e106abb1347eaf4dc87cbcf00f9e57aed6acffc5a1e70c4c8fb59e8a5" Oct 30 22:28:33.686474 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:33.686431 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:33.856835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:33.856803 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:34.218641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:34.218601 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:34.862541 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:34.862508 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:35.236594 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:35.236512 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:36.735893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:36.735859 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:39.551719 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:39.551687 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:39.729497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:39.729463 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:39.909476 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:39.909438 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:40.371928 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:28:40.371894 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:28:42.851001 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:42.850942 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:43.030928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:43.030898 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:46.210223 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:46.210188 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:46.378224 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:46.378192 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:46.529069 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:46.528971 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:51.053262 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:51.053221 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt"] Oct 30 22:28:51.102693 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:51.102662 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-7sbdt"] Oct 30 22:28:51.221011 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:51.220947 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:51.957969 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:51.957927 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a3caea-974d-4618-86ec-74a237e3cc5e" path="/var/lib/kubelet/pods/d4a3caea-974d-4618-86ec-74a237e3cc5e/volumes" Oct 30 22:28:56.179928 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:28:56.179880 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:01.120669 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:01.120637 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:03.046830 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:03.046796 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz"] Oct 30 22:29:03.096018 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:03.095987 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-g9lxz"] Oct 30 22:29:03.951180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:03.951144 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61eab7e2-895d-4d0a-966e-20b860613890" path="/var/lib/kubelet/pods/61eab7e2-895d-4d0a-966e-20b860613890/volumes" Oct 30 22:29:04.392587 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:04.392552 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:05.187145 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:05.187103 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:05.355836 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:05.355801 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:06.398361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:06.398317 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:08.995581 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:08.995540 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:10.371884 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:29:10.371848 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:29:11.277799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:11.277764 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:16.166189 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:16.166151 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:21.075109 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:21.075063 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:25.974863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:25.974826 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:30.778997 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:30.778961 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:35.681051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:35.681014 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:37.082802 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:37.082756 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:39.060814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:39.060778 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv"] Oct 30 22:29:39.110251 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:39.110220 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-7lrqv"] Oct 30 22:29:39.951014 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:39.950972 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59783c93-2c17-45c0-accd-18cf0c8962b5" path="/var/lib/kubelet/pods/59783c93-2c17-45c0-accd-18cf0c8962b5/volumes" Oct 30 22:29:40.372091 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:29:40.372057 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:29:40.542290 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:40.542240 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:45.368569 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:45.368515 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:49.054897 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:49.054824 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr"] Oct 30 22:29:49.105140 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:49.105107 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5kjlr"] Oct 30 22:29:49.958159 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:49.958125 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba" path="/var/lib/kubelet/pods/b9bbeb84-ea0b-451b-b2d8-545fab5bc6ba/volumes" Oct 30 22:29:50.191040 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:50.191006 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:54.339530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.339491 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85"] Oct 30 22:29:54.339948 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.339875 5508 cpu_manager.go:414] "containerMap: removing container" podUID="b106d7cc-0c83-4238-814f-ee70366c7620" containerName="revoke-keys" Oct 30 22:29:54.339948 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.339886 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b106d7cc-0c83-4238-814f-ee70366c7620" containerName="revoke-keys" Oct 30 22:29:54.340025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.339976 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b106d7cc-0c83-4238-814f-ee70366c7620" containerName="revoke-keys" Oct 30 22:29:54.343398 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.343381 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.398124 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.398088 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:29:54.437789 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.437751 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.437789 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.437794 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.438050 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.437971 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pxcj\" (UniqueName: \"kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.447193 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.447160 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:29:54.447361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.447177 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:29:54.447361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.447248 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:29:54.449196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.449175 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85"] Oct 30 22:29:54.538761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.538730 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.538909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.538766 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.538909 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.538835 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pxcj\" (UniqueName: \"kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.539453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.539435 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.541258 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.541239 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.595970 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.595910 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pxcj\" (UniqueName: \"kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj\") pod \"maas-api-revoke-sub-c0a67cb3-tsf85\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:54.656917 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:54.656884 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:55.028654 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:55.028627 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85"] Oct 30 22:29:55.031391 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:29:55.031363 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaa7b1f_3998_4ee7_b54d_8dc3d404303f.slice/crio-6c558cfa4eee9abfd5fcd44a50950c582d74d0a9b88b28a62e69048a23f3bf66 WatchSource:0}: Error finding container 6c558cfa4eee9abfd5fcd44a50950c582d74d0a9b88b28a62e69048a23f3bf66: Status 404 returned error can't find the container with id 6c558cfa4eee9abfd5fcd44a50950c582d74d0a9b88b28a62e69048a23f3bf66 Oct 30 22:29:55.069426 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:55.069402 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:56.026801 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:56.026709 5508 generic.go:356] "Generic (PLEG): container finished" podID="eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" containerID="9d93b007830fe7640582d4bc3f17e4d2bbefbeadc1678b6b7be534668b57dc2e" exitCode=0 Oct 30 22:29:56.026801 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:56.026786 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" event={"ID":"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f","Type":"ContainerDied","Data":"9d93b007830fe7640582d4bc3f17e4d2bbefbeadc1678b6b7be534668b57dc2e"} Oct 30 22:29:56.027197 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:56.026838 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" event={"ID":"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f","Type":"ContainerStarted","Data":"6c558cfa4eee9abfd5fcd44a50950c582d74d0a9b88b28a62e69048a23f3bf66"} Oct 30 22:29:57.169372 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.169344 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:57.267532 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.267499 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca\") pod \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " Oct 30 22:29:57.267713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.267558 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca\") pod \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " Oct 30 22:29:57.267713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.267629 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj\" (UniqueName: \"kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj\") pod \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\" (UID: \"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f\") " Oct 30 22:29:57.267906 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.267878 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca" pod "eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" (UID: "eaaa7b1f-3998-4ee7-b54d-8dc3d404303f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:57.268008 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.267995 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.269834 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.269809 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca" pod "eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" (UID: "eaaa7b1f-3998-4ee7-b54d-8dc3d404303f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:57.270077 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.270044 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj" pod "eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" (UID: "eaaa7b1f-3998-4ee7-b54d-8dc3d404303f"). InnerVolumeSpecName "kube-api-access-2pxcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:57.368795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.368766 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:57.368795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:57.368794 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pxcj\" (UniqueName: \"kubernetes.io/projected/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f-kube-api-access-2pxcj\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:29:58.038671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:58.038646 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" Oct 30 22:29:58.038831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:58.038683 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85" event={"ID":"eaaa7b1f-3998-4ee7-b54d-8dc3d404303f","Type":"ContainerDied","Data":"6c558cfa4eee9abfd5fcd44a50950c582d74d0a9b88b28a62e69048a23f3bf66"} Oct 30 22:29:58.038831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:58.038732 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c558cfa4eee9abfd5fcd44a50950c582d74d0a9b88b28a62e69048a23f3bf66" Oct 30 22:29:59.914569 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:29:59.914526 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:00.160165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.160126 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq"] Oct 30 22:30:00.160564 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.160550 5508 cpu_manager.go:414] "containerMap: removing container" podUID="eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" containerName="revoke-keys" Oct 30 22:30:00.160613 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.160566 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" containerName="revoke-keys" Oct 30 22:30:00.160649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.160643 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" containerName="revoke-keys" Oct 30 22:30:00.163905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.163884 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.216352 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.216292 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:30:00.265684 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.265647 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.265874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.265692 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.267710 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.267674 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq"] Oct 30 22:30:00.297361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.297328 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-l6plq\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.297572 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.297404 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zfl9\" (UniqueName: \"kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9\") pod \"maas-api-key-cleanup-29889990-l6plq\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.398519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.398487 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-l6plq\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.398727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.398554 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfl9\" (UniqueName: \"kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9\") pod \"maas-api-key-cleanup-29889990-l6plq\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.399240 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.399213 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-l6plq\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.455709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.455645 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfl9\" (UniqueName: \"kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9\") pod \"maas-api-key-cleanup-29889990-l6plq\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.477446 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.477425 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:00.851531 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:00.851505 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq"] Oct 30 22:30:00.854079 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:30:00.854047 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6ffc23_7533_424d_9779_a9dd2ebb8114.slice/crio-52c259a0ee09248bf8f9c501007044b5f9c2b5f2103e4ffe0225ea9b0696c080 WatchSource:0}: Error finding container 52c259a0ee09248bf8f9c501007044b5f9c2b5f2103e4ffe0225ea9b0696c080: Status 404 returned error can't find the container with id 52c259a0ee09248bf8f9c501007044b5f9c2b5f2103e4ffe0225ea9b0696c080 Oct 30 22:30:01.056714 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:01.056670 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" event={"ID":"3e6ffc23-7533-424d-9779-a9dd2ebb8114","Type":"ContainerStarted","Data":"52c259a0ee09248bf8f9c501007044b5f9c2b5f2103e4ffe0225ea9b0696c080"} Oct 30 22:30:02.064192 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:02.064100 5508 generic.go:356] "Generic (PLEG): container finished" podID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerID="16706033faf56d7b724ae8792931561698278b3e68aaf3c6b1a9d1f0fec363d0" exitCode=60 Oct 30 22:30:02.064192 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:02.064159 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" event={"ID":"3e6ffc23-7533-424d-9779-a9dd2ebb8114","Type":"ContainerDied","Data":"16706033faf56d7b724ae8792931561698278b3e68aaf3c6b1a9d1f0fec363d0"} Oct 30 22:30:02.064685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:02.064533 5508 scope.go:122] "RemoveContainer" containerID="16706033faf56d7b724ae8792931561698278b3e68aaf3c6b1a9d1f0fec363d0" Oct 30 22:30:03.071756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:03.071720 5508 generic.go:356] "Generic (PLEG): container finished" podID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerID="f2054bed453d6851a55aaec73875e5aa676fb347afc47b332ad633ec8ed2039a" exitCode=60 Oct 30 22:30:03.072239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:03.071813 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" event={"ID":"3e6ffc23-7533-424d-9779-a9dd2ebb8114","Type":"ContainerDied","Data":"f2054bed453d6851a55aaec73875e5aa676fb347afc47b332ad633ec8ed2039a"} Oct 30 22:30:03.072239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:03.071882 5508 scope.go:122] "RemoveContainer" containerID="16706033faf56d7b724ae8792931561698278b3e68aaf3c6b1a9d1f0fec363d0" Oct 30 22:30:03.072239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:03.072155 5508 scope.go:122] "RemoveContainer" containerID="f2054bed453d6851a55aaec73875e5aa676fb347afc47b332ad633ec8ed2039a" Oct 30 22:30:03.072525 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:30:03.072504 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-l6plq_odh-ai-gateway-infra(3e6ffc23-7533-424d-9779-a9dd2ebb8114)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" Oct 30 22:30:04.079924 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:04.079898 5508 scope.go:122] "RemoveContainer" containerID="f2054bed453d6851a55aaec73875e5aa676fb347afc47b332ad633ec8ed2039a" Oct 30 22:30:04.080321 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:30:04.080145 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-l6plq_odh-ai-gateway-infra(3e6ffc23-7533-424d-9779-a9dd2ebb8114)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" Oct 30 22:30:04.777795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:04.777762 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:06.051775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.051721 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:06.519626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.519591 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc"] Oct 30 22:30:06.523046 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.523027 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.561238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.561209 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpdr\" (UniqueName: \"kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.561404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.561263 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.561404 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.561382 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.577270 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.577237 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:06.628461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.628427 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc"] Oct 30 22:30:06.662237 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.662204 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.662466 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.662346 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpdr\" (UniqueName: \"kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.662466 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.662409 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.663073 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.663052 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.664828 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.664802 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.718815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.718778 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpdr\" (UniqueName: \"kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:06.835900 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:06.835866 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:07.208543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:07.208518 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc"] Oct 30 22:30:07.211187 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:30:07.211156 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c38aaaf_b32e_4cbb_9143_616c4454463c.slice/crio-65fb9ab7823aa450809ae2b458a1b52e4d6fdec01f41b5b8ee3ef30ddf312bd8 WatchSource:0}: Error finding container 65fb9ab7823aa450809ae2b458a1b52e4d6fdec01f41b5b8ee3ef30ddf312bd8: Status 404 returned error can't find the container with id 65fb9ab7823aa450809ae2b458a1b52e4d6fdec01f41b5b8ee3ef30ddf312bd8 Oct 30 22:30:08.102465 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:08.102430 5508 generic.go:356] "Generic (PLEG): container finished" podID="4c38aaaf-b32e-4cbb-9143-616c4454463c" containerID="a2cdcb8914603d99081f1308db122177fb89c6279ffd93d1cf76c919e87cae42" exitCode=0 Oct 30 22:30:08.102630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:08.102489 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" event={"ID":"4c38aaaf-b32e-4cbb-9143-616c4454463c","Type":"ContainerDied","Data":"a2cdcb8914603d99081f1308db122177fb89c6279ffd93d1cf76c919e87cae42"} Oct 30 22:30:08.102630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:08.102529 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" event={"ID":"4c38aaaf-b32e-4cbb-9143-616c4454463c","Type":"ContainerStarted","Data":"65fb9ab7823aa450809ae2b458a1b52e4d6fdec01f41b5b8ee3ef30ddf312bd8"} Oct 30 22:30:09.251082 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.251053 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:09.288529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.288486 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr\" (UniqueName: \"kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr\") pod \"4c38aaaf-b32e-4cbb-9143-616c4454463c\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " Oct 30 22:30:09.288685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.288631 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca\") pod \"4c38aaaf-b32e-4cbb-9143-616c4454463c\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " Oct 30 22:30:09.288727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.288687 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca\") pod \"4c38aaaf-b32e-4cbb-9143-616c4454463c\" (UID: \"4c38aaaf-b32e-4cbb-9143-616c4454463c\") " Oct 30 22:30:09.288975 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.288950 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca" pod "4c38aaaf-b32e-4cbb-9143-616c4454463c" (UID: "4c38aaaf-b32e-4cbb-9143-616c4454463c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:09.290705 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.290676 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr" pod "4c38aaaf-b32e-4cbb-9143-616c4454463c" (UID: "4c38aaaf-b32e-4cbb-9143-616c4454463c"). InnerVolumeSpecName "kube-api-access-8fpdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:09.290793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.290756 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca" pod "4c38aaaf-b32e-4cbb-9143-616c4454463c" (UID: "4c38aaaf-b32e-4cbb-9143-616c4454463c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:09.390112 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.390036 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:09.390112 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.390064 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c38aaaf-b32e-4cbb-9143-616c4454463c-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:09.390112 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.390078 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fpdr\" (UniqueName: \"kubernetes.io/projected/4c38aaaf-b32e-4cbb-9143-616c4454463c-kube-api-access-8fpdr\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:09.960752 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:09.960722 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:10.114583 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:10.114552 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" Oct 30 22:30:10.114788 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:10.114558 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc" event={"ID":"4c38aaaf-b32e-4cbb-9143-616c4454463c","Type":"ContainerDied","Data":"65fb9ab7823aa450809ae2b458a1b52e4d6fdec01f41b5b8ee3ef30ddf312bd8"} Oct 30 22:30:10.114788 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:10.114666 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65fb9ab7823aa450809ae2b458a1b52e4d6fdec01f41b5b8ee3ef30ddf312bd8" Oct 30 22:30:10.373074 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:30:10.373031 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:30:11.009780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:11.009751 5508 scope.go:122] "RemoveContainer" containerID="12417291024ef1ef6951767d8e5f7c13dc81f649f8283b32b1faf95ad2ac842f" Oct 30 22:30:11.018643 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:11.018622 5508 scope.go:122] "RemoveContainer" containerID="ccd0ca1cc97db8fab5a885cf92008ecf6e09cd2fb2834b4f2127648d40eefe64" Oct 30 22:30:11.027671 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:11.027647 5508 scope.go:122] "RemoveContainer" containerID="6b43ab779d6546cee3cac5b137543d5b507bf2a39bdadbf69a26b8f6e196ccca" Oct 30 22:30:11.085774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:11.085733 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:15.067882 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:15.067850 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:16.945787 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:16.945758 5508 scope.go:122] "RemoveContainer" containerID="f2054bed453d6851a55aaec73875e5aa676fb347afc47b332ad633ec8ed2039a" Oct 30 22:30:18.161287 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:18.161252 5508 generic.go:356] "Generic (PLEG): container finished" podID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerID="a0f6ca2d90a3e9bc3f892b4573d9ef9439579442debd70aa46c6b63a9ddf4c7f" exitCode=60 Oct 30 22:30:18.161718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:18.161348 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" event={"ID":"3e6ffc23-7533-424d-9779-a9dd2ebb8114","Type":"ContainerDied","Data":"a0f6ca2d90a3e9bc3f892b4573d9ef9439579442debd70aa46c6b63a9ddf4c7f"} Oct 30 22:30:18.161718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:18.161410 5508 scope.go:122] "RemoveContainer" containerID="f2054bed453d6851a55aaec73875e5aa676fb347afc47b332ad633ec8ed2039a" Oct 30 22:30:18.162072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:18.162043 5508 scope.go:122] "RemoveContainer" containerID="a0f6ca2d90a3e9bc3f892b4573d9ef9439579442debd70aa46c6b63a9ddf4c7f" Oct 30 22:30:18.162768 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:30:18.162728 5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-l6plq_odh-ai-gateway-infra(3e6ffc23-7533-424d-9779-a9dd2ebb8114)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" Oct 30 22:30:18.382993 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:18.382953 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:18.547699 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:18.547608 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:19.285603 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.285564 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq"] Oct 30 22:30:19.431045 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.431020 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:19.498081 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.497848 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9\" (UniqueName: \"kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9\") pod \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " Oct 30 22:30:19.498081 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.497934 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca\") pod \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\" (UID: \"3e6ffc23-7533-424d-9779-a9dd2ebb8114\") " Oct 30 22:30:19.500380 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.499069 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca" pod "3e6ffc23-7533-424d-9779-a9dd2ebb8114" (UID: "3e6ffc23-7533-424d-9779-a9dd2ebb8114"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.515596 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.515557 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9" pod "3e6ffc23-7533-424d-9779-a9dd2ebb8114" (UID: "3e6ffc23-7533-424d-9779-a9dd2ebb8114"). InnerVolumeSpecName "kube-api-access-7zfl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.599061 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.599026 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zfl9\" (UniqueName: \"kubernetes.io/projected/3e6ffc23-7533-424d-9779-a9dd2ebb8114-kube-api-access-7zfl9\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.599061 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.599056 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6ffc23-7533-424d-9779-a9dd2ebb8114-maas-api-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.869818 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:19.869720 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:20.174701 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:20.174618 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" Oct 30 22:30:20.174701 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:20.174644 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq" event={"ID":"3e6ffc23-7533-424d-9779-a9dd2ebb8114","Type":"ContainerDied","Data":"52c259a0ee09248bf8f9c501007044b5f9c2b5f2103e4ffe0225ea9b0696c080"} Oct 30 22:30:20.174947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:20.174707 5508 scope.go:122] "RemoveContainer" containerID="a0f6ca2d90a3e9bc3f892b4573d9ef9439579442debd70aa46c6b63a9ddf4c7f" Oct 30 22:30:20.335883 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:20.335849 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq"] Oct 30 22:30:20.384925 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:20.384896 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-l6plq"] Oct 30 22:30:21.950801 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:21.950773 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" path="/var/lib/kubelet/pods/3e6ffc23-7533-424d-9779-a9dd2ebb8114/volumes" Oct 30 22:30:24.045799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:24.045763 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw"] Oct 30 22:30:24.096530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:24.096484 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-r9bgw"] Oct 30 22:30:24.186798 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:24.186760 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:25.744845 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:25.744809 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:25.951717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:25.951678 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9290f7ec-691e-49e7-8519-c6e8928d7bb2" path="/var/lib/kubelet/pods/9290f7ec-691e-49e7-8519-c6e8928d7bb2/volumes" Oct 30 22:30:26.025998 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:26.025922 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:26.332842 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:26.332801 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc"] Oct 30 22:30:26.382217 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:26.382163 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9513bb-l5hjc"] Oct 30 22:30:26.752186 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:26.752097 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:27.950920 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:27.950888 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c38aaaf-b32e-4cbb-9143-616c4454463c" path="/var/lib/kubelet/pods/4c38aaaf-b32e-4cbb-9143-616c4454463c/volumes" Oct 30 22:30:32.638683 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:32.638644 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:32.823518 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:32.823479 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:33.088347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:33.088273 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:34.385020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.384988 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk"] Oct 30 22:30:34.434154 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.434124 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nqjbk"] Oct 30 22:30:34.508619 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.508582 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g"] Oct 30 22:30:34.508963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.508951 5508 cpu_manager.go:414] "containerMap: removing container" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.509020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.508964 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.509020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.508979 5508 cpu_manager.go:414] "containerMap: removing container" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.509020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.508985 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.509020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.509008 5508 cpu_manager.go:414] "containerMap: removing container" podUID="4c38aaaf-b32e-4cbb-9143-616c4454463c" containerName="revoke-keys" Oct 30 22:30:34.509020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.509013 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c38aaaf-b32e-4cbb-9143-616c4454463c" containerName="revoke-keys" Oct 30 22:30:34.509174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.509090 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c38aaaf-b32e-4cbb-9143-616c4454463c" containerName="revoke-keys" Oct 30 22:30:34.509174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.509102 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.509174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.509110 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.509174 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.509116 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:30:34.513823 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.513802 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.568244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.567895 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:34.568244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.568147 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:34.568244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.568186 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:30:34.568244 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.568205 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:30:34.570194 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.570167 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g"] Oct 30 22:30:34.651641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.651561 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.651641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.651631 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzns\" (UniqueName: \"kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.651865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.651753 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.752863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.752831 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzns\" (UniqueName: \"kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.753024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.752932 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.753024 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.752975 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.753621 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.753586 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.755499 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.755482 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.809626 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.809599 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzns\" (UniqueName: \"kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns\") pod \"maas-api-revoke-sub-3d4156f1-t626g\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:34.827425 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:34.827404 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:35.200770 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:35.200735 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g"] Oct 30 22:30:35.203916 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:30:35.203887 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd03ccbc_a80e_4751_9e43_2b07b890bba3.slice/crio-102a1a8631dbe85e49f67ec9725254b0bc6c30d508b44b2598fe349b1ac8a6e6 WatchSource:0}: Error finding container 102a1a8631dbe85e49f67ec9725254b0bc6c30d508b44b2598fe349b1ac8a6e6: Status 404 returned error can't find the container with id 102a1a8631dbe85e49f67ec9725254b0bc6c30d508b44b2598fe349b1ac8a6e6 Oct 30 22:30:35.257801 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:35.257774 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" event={"ID":"cd03ccbc-a80e-4751-9e43-2b07b890bba3","Type":"ContainerStarted","Data":"102a1a8631dbe85e49f67ec9725254b0bc6c30d508b44b2598fe349b1ac8a6e6"} Oct 30 22:30:35.951589 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:35.951554 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b106d7cc-0c83-4238-814f-ee70366c7620" path="/var/lib/kubelet/pods/b106d7cc-0c83-4238-814f-ee70366c7620/volumes" Oct 30 22:30:36.264294 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:36.264210 5508 generic.go:356] "Generic (PLEG): container finished" podID="cd03ccbc-a80e-4751-9e43-2b07b890bba3" containerID="d851e12509d91236c5c2394c95d3c888753e8ee8d4146ff946c926d9b8fd6875" exitCode=0 Oct 30 22:30:36.264461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:36.264296 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" event={"ID":"cd03ccbc-a80e-4751-9e43-2b07b890bba3","Type":"ContainerDied","Data":"d851e12509d91236c5c2394c95d3c888753e8ee8d4146ff946c926d9b8fd6875"} Oct 30 22:30:37.410262 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.410238 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:37.419766 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.419734 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:37.582821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.582793 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns\" (UniqueName: \"kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns\") pod \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " Oct 30 22:30:37.583015 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.582994 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca\") pod \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " Oct 30 22:30:37.583072 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.583052 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca\") pod \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\" (UID: \"cd03ccbc-a80e-4751-9e43-2b07b890bba3\") " Oct 30 22:30:37.583461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.583429 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca" pod "cd03ccbc-a80e-4751-9e43-2b07b890bba3" (UID: "cd03ccbc-a80e-4751-9e43-2b07b890bba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:37.585071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.585045 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns" pod "cd03ccbc-a80e-4751-9e43-2b07b890bba3" (UID: "cd03ccbc-a80e-4751-9e43-2b07b890bba3"). InnerVolumeSpecName "kube-api-access-nlzns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:37.585179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.585149 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca" pod "cd03ccbc-a80e-4751-9e43-2b07b890bba3" (UID: "cd03ccbc-a80e-4751-9e43-2b07b890bba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:37.684685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.684654 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzns\" (UniqueName: \"kubernetes.io/projected/cd03ccbc-a80e-4751-9e43-2b07b890bba3-kube-api-access-nlzns\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:37.684685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.684683 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:37.684685 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:37.684693 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd03ccbc-a80e-4751-9e43-2b07b890bba3-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:38.281105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:38.281021 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" Oct 30 22:30:38.281105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:38.281050 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g" event={"ID":"cd03ccbc-a80e-4751-9e43-2b07b890bba3","Type":"ContainerDied","Data":"102a1a8631dbe85e49f67ec9725254b0bc6c30d508b44b2598fe349b1ac8a6e6"} Oct 30 22:30:38.281105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:38.281098 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="102a1a8631dbe85e49f67ec9725254b0bc6c30d508b44b2598fe349b1ac8a6e6" Oct 30 22:30:40.371364 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:30:40.371326 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:30:47.833666 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:47.833626 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:48.457980 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:48.457940 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:52.045539 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:52.045501 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf"] Oct 30 22:30:52.094865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:52.094830 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-66cdf"] Oct 30 22:30:53.951272 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:53.951236 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1651be40-be6c-498b-9bcb-7649a8cecae3" path="/var/lib/kubelet/pods/1651be40-be6c-498b-9bcb-7649a8cecae3/volumes" Oct 30 22:30:54.371677 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:30:54.371638 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:10.371898 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:31:10.371861 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:31:11.056739 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:11.056707 5508 scope.go:122] "RemoveContainer" containerID="4cc7574fc60996e3c6eec0360094deab954789289a4d2097e68612f09a97e982" Oct 30 22:31:11.066050 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:11.066026 5508 scope.go:122] "RemoveContainer" containerID="440e6a8d1db5b3c1d2570d98d28543459c2bf6daa7dc55af320f0a6ef6592aff" Oct 30 22:31:23.048958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:23.048899 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85"] Oct 30 22:31:23.095957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:23.095928 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-v6h85"] Oct 30 22:31:23.953408 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:23.953368 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7f74f6-7cac-440c-8883-2e1f1c26c7fa" path="/var/lib/kubelet/pods/cc7f74f6-7cac-440c-8883-2e1f1c26c7fa/volumes" Oct 30 22:31:37.774431 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:37.774391 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:40.372647 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:31:40.372614 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:31:43.177995 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:43.177959 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.352585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:43.352554 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:44.047278 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:44.047239 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2"] Oct 30 22:31:44.096214 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:44.096181 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-mx4s2"] Oct 30 22:31:45.950953 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:31:45.950919 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d" path="/var/lib/kubelet/pods/4cc16e57-2e09-4cdc-8a88-4f6b6fd8514d/volumes" Oct 30 22:32:10.371342 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:32:10.371286 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:32:11.095924 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:11.095892 5508 scope.go:122] "RemoveContainer" containerID="0f21c36afc889c8636288668167f7019b2352d50f76f6629e60e956448c6baca" Oct 30 22:32:11.105731 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:11.105706 5508 scope.go:122] "RemoveContainer" containerID="dce66e7741b581e5f8bb71c67fde1b0765181b60433c124ab1966477dd46d41a" Oct 30 22:32:23.269455 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:23.269411 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:32:23.269937 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:23.269479 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:32:25.284478 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.284433 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt"] Oct 30 22:32:25.284991 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.284971 5508 cpu_manager.go:414] "containerMap: removing container" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:32:25.285397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.284993 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6ffc23-7533-424d-9779-a9dd2ebb8114" containerName="cleanup" Oct 30 22:32:25.285397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.285011 5508 cpu_manager.go:414] "containerMap: removing container" podUID="cd03ccbc-a80e-4751-9e43-2b07b890bba3" containerName="revoke-keys" Oct 30 22:32:25.285397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.285017 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd03ccbc-a80e-4751-9e43-2b07b890bba3" containerName="revoke-keys" Oct 30 22:32:25.285397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.285101 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd03ccbc-a80e-4751-9e43-2b07b890bba3" containerName="revoke-keys" Oct 30 22:32:25.289745 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.289726 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.343450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.343429 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:25.392450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.392418 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:25.392624 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.392466 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:25.393343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.393024 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:32:25.397339 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.396514 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt"] Oct 30 22:32:25.466299 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.466264 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.466522 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.466376 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.466522 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.466454 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hth\" (UniqueName: \"kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.567232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.567140 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hth\" (UniqueName: \"kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.567232 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.567223 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.567498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.567249 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.567874 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.567853 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.569786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.569762 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.623607 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.623583 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hth\" (UniqueName: \"kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth\") pod \"maas-api-revoke-sub-d576e68b-7c7kt\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:25.902741 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:25.902708 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:26.274687 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:26.274664 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt"] Oct 30 22:32:26.277351 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:32:26.277325 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeed1c66_a1ed_44c0_b621_244906ea15c8.slice/crio-2e701f9bc5f7cffb567591845fc921cbaa95918227b2a6a2c8f467701fa98a4e WatchSource:0}: Error finding container 2e701f9bc5f7cffb567591845fc921cbaa95918227b2a6a2c8f467701fa98a4e: Status 404 returned error can't find the container with id 2e701f9bc5f7cffb567591845fc921cbaa95918227b2a6a2c8f467701fa98a4e Oct 30 22:32:26.849239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:26.849156 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" event={"ID":"aeed1c66-a1ed-44c0-b621-244906ea15c8","Type":"ContainerStarted","Data":"f53f50728d2c8ab6434f1bfce9f70c22e06c8e62d52977888950b95bcd9577ef"} Oct 30 22:32:26.849239 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:26.849216 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" event={"ID":"aeed1c66-a1ed-44c0-b621-244906ea15c8","Type":"ContainerStarted","Data":"2e701f9bc5f7cffb567591845fc921cbaa95918227b2a6a2c8f467701fa98a4e"} Oct 30 22:32:26.964203 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:26.964152 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" podStartSLOduration=1.964140014 podStartE2EDuration="1.964140014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:26.962385633 +0000 UTC m=+2957.151923335" watchObservedRunningTime="2026-10-30 22:32:26.964140014 +0000 UTC m=+2957.153677688" Oct 30 22:32:27.855718 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:27.855682 5508 generic.go:356] "Generic (PLEG): container finished" podID="aeed1c66-a1ed-44c0-b621-244906ea15c8" containerID="f53f50728d2c8ab6434f1bfce9f70c22e06c8e62d52977888950b95bcd9577ef" exitCode=0 Oct 30 22:32:27.856105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:27.855769 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" event={"ID":"aeed1c66-a1ed-44c0-b621-244906ea15c8","Type":"ContainerDied","Data":"f53f50728d2c8ab6434f1bfce9f70c22e06c8e62d52977888950b95bcd9577ef"} Oct 30 22:32:28.998447 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:28.998420 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:29.102585 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.102559 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca\") pod \"aeed1c66-a1ed-44c0-b621-244906ea15c8\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " Oct 30 22:32:29.102717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.102598 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca\") pod \"aeed1c66-a1ed-44c0-b621-244906ea15c8\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " Oct 30 22:32:29.102764 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.102718 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth\" (UniqueName: \"kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth\") pod \"aeed1c66-a1ed-44c0-b621-244906ea15c8\" (UID: \"aeed1c66-a1ed-44c0-b621-244906ea15c8\") " Oct 30 22:32:29.102978 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.102954 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca" pod "aeed1c66-a1ed-44c0-b621-244906ea15c8" (UID: "aeed1c66-a1ed-44c0-b621-244906ea15c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:29.104865 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.104837 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca" pod "aeed1c66-a1ed-44c0-b621-244906ea15c8" (UID: "aeed1c66-a1ed-44c0-b621-244906ea15c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:29.104970 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.104841 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth" pod "aeed1c66-a1ed-44c0-b621-244906ea15c8" (UID: "aeed1c66-a1ed-44c0-b621-244906ea15c8"). InnerVolumeSpecName "kube-api-access-h2hth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:29.204012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.203980 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2hth\" (UniqueName: \"kubernetes.io/projected/aeed1c66-a1ed-44c0-b621-244906ea15c8-kube-api-access-h2hth\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:29.204012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.204008 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:29.204180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.204017 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeed1c66-a1ed-44c0-b621-244906ea15c8-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:29.868180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.868153 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" Oct 30 22:32:29.868361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.868152 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt" event={"ID":"aeed1c66-a1ed-44c0-b621-244906ea15c8","Type":"ContainerDied","Data":"2e701f9bc5f7cffb567591845fc921cbaa95918227b2a6a2c8f467701fa98a4e"} Oct 30 22:32:29.868361 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:29.868279 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e701f9bc5f7cffb567591845fc921cbaa95918227b2a6a2c8f467701fa98a4e" Oct 30 22:32:37.187088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:37.187049 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:38.117695 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:38.117656 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:40.372622 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:32:40.372588 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:32:54.044851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.044754 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt"] Oct 30 22:32:54.093887 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.093859 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cqrqt"] Oct 30 22:32:54.225686 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.225650 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd"] Oct 30 22:32:54.226164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.226147 5508 cpu_manager.go:414] "containerMap: removing container" podUID="aeed1c66-a1ed-44c0-b621-244906ea15c8" containerName="revoke-keys" Oct 30 22:32:54.226282 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.226167 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeed1c66-a1ed-44c0-b621-244906ea15c8" containerName="revoke-keys" Oct 30 22:32:54.226391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.226320 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeed1c66-a1ed-44c0-b621-244906ea15c8" containerName="revoke-keys" Oct 30 22:32:54.229402 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.229382 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.281649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.281622 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:32:54.281649 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.281635 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:32:54.281806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.281690 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:32:54.281806 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.281716 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:32:54.286694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.286671 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd"] Oct 30 22:32:54.348259 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.348219 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjd7v\" (UniqueName: \"kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.348479 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.348411 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.348479 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.348451 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.449808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.449776 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.449965 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.449822 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.449965 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.449868 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjd7v\" (UniqueName: \"kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.450536 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.450518 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.452182 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.452162 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.506717 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.506686 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjd7v\" (UniqueName: \"kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v\") pod \"maas-api-revoke-sub-d2bc93bc-kkkcd\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.541675 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.541645 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:54.912247 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:54.912219 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd"] Oct 30 22:32:54.914907 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:32:54.914879 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1079f7_e302_4965_86bd_ba7b8b7e81ca.slice/crio-faa3f1703f40953e0797b140810119671d7eadbf96bfe5a5f38c0e11af116f48 WatchSource:0}: Error finding container faa3f1703f40953e0797b140810119671d7eadbf96bfe5a5f38c0e11af116f48: Status 404 returned error can't find the container with id faa3f1703f40953e0797b140810119671d7eadbf96bfe5a5f38c0e11af116f48 Oct 30 22:32:55.007976 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:55.007944 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" event={"ID":"4d1079f7-e302-4965-86bd-ba7b8b7e81ca","Type":"ContainerStarted","Data":"faa3f1703f40953e0797b140810119671d7eadbf96bfe5a5f38c0e11af116f48"} Oct 30 22:32:55.955382 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:55.955346 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e116f826-e5e9-4f7c-8274-0c7214e38972" path="/var/lib/kubelet/pods/e116f826-e5e9-4f7c-8274-0c7214e38972/volumes" Oct 30 22:32:56.013913 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:56.013836 5508 generic.go:356] "Generic (PLEG): container finished" podID="4d1079f7-e302-4965-86bd-ba7b8b7e81ca" containerID="b5d4416d8911be8b95092559e08fc5eecad7a44d8ba1d8e5af862a542acac7ec" exitCode=0 Oct 30 22:32:56.014054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:56.013927 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" event={"ID":"4d1079f7-e302-4965-86bd-ba7b8b7e81ca","Type":"ContainerDied","Data":"b5d4416d8911be8b95092559e08fc5eecad7a44d8ba1d8e5af862a542acac7ec"} Oct 30 22:32:57.153327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.153287 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:57.275028 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.274940 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca\") pod \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " Oct 30 22:32:57.275203 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.275074 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v\" (UniqueName: \"kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v\") pod \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " Oct 30 22:32:57.275274 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.275198 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca\") pod \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\" (UID: \"4d1079f7-e302-4965-86bd-ba7b8b7e81ca\") " Oct 30 22:32:57.275402 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.275381 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca" pod "4d1079f7-e302-4965-86bd-ba7b8b7e81ca" (UID: "4d1079f7-e302-4965-86bd-ba7b8b7e81ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:57.275591 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.275572 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:57.277368 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.277334 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v" pod "4d1079f7-e302-4965-86bd-ba7b8b7e81ca" (UID: "4d1079f7-e302-4965-86bd-ba7b8b7e81ca"). InnerVolumeSpecName "kube-api-access-kjd7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:57.277486 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.277401 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca" pod "4d1079f7-e302-4965-86bd-ba7b8b7e81ca" (UID: "4d1079f7-e302-4965-86bd-ba7b8b7e81ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:57.376740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.376695 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjd7v\" (UniqueName: \"kubernetes.io/projected/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-kube-api-access-kjd7v\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:57.376740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:57.376736 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1079f7-e302-4965-86bd-ba7b8b7e81ca-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:32:58.024693 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:58.024667 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" Oct 30 22:32:58.024831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:58.024703 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd" event={"ID":"4d1079f7-e302-4965-86bd-ba7b8b7e81ca","Type":"ContainerDied","Data":"faa3f1703f40953e0797b140810119671d7eadbf96bfe5a5f38c0e11af116f48"} Oct 30 22:32:58.024831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:32:58.024755 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa3f1703f40953e0797b140810119671d7eadbf96bfe5a5f38c0e11af116f48" Oct 30 22:33:05.045133 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:05.045089 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl"] Oct 30 22:33:05.094209 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:05.094172 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-4qldl"] Oct 30 22:33:05.952097 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:05.952062 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ecdfb2-087e-4113-ad2e-015bf4da26ef" path="/var/lib/kubelet/pods/48ecdfb2-087e-4113-ad2e-015bf4da26ef/volumes" Oct 30 22:33:06.508959 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:06.508924 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:07.240955 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:07.240920 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:07.522319 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:07.522224 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:10.372228 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:33:10.372192 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:33:10.902948 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:10.902913 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:11.067902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:11.067839 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:11.129690 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:11.129656 5508 scope.go:122] "RemoveContainer" containerID="ac63862b273465a2d380c3a483c4ed77aaaeee7a065789fabc6111a4cd91bdad" Oct 30 22:33:11.146139 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:11.146114 5508 scope.go:122] "RemoveContainer" containerID="40a66a31a2676d1952b08f77358fcfde8b2831dd3d38ed5d91279aa07f9f60f3" Oct 30 22:33:11.154292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:11.154266 5508 scope.go:122] "RemoveContainer" containerID="727075d02f542cca44b724e220a40bac0094061e80daa3bd1a9beb4a57f49f57" Oct 30 22:33:11.231774 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:11.231741 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:14.113575 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:14.113540 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:14.290795 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:14.290758 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:17.430744 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:17.430707 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:17.589756 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:17.589720 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:17.744054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:17.743943 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:22.393179 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:22.393143 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:27.502451 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:27.502414 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:32.462462 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:32.462427 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:35.832263 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:35.832211 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:36.553494 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:36.553440 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:36.719237 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:36.719199 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:37.711493 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:37.711451 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:38.451525 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:38.451491 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:39.028893 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:39.028857 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:40.373255 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:33:40.373218 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:33:42.571557 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:42.571518 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:47.494130 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:47.494068 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:52.268435 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:52.268401 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:57.173860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:33:57.173828 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:01.979597 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:01.979562 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:06.870120 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:06.870086 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:10.372283 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:34:10.372241 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:34:11.181229 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:11.181200 5508 scope.go:122] "RemoveContainer" containerID="5832a97d8b2bc7a633e981823336d925827c84f8f143bda93946c50eb3a047c8" Oct 30 22:34:11.190009 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:11.189980 5508 scope.go:122] "RemoveContainer" containerID="043458562cfbe7b3bbb74032015a83737fcd6a74ed5cea6771162b10577b30f2" Oct 30 22:34:11.198336 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:11.198291 5508 scope.go:122] "RemoveContainer" containerID="410ab08aa7ab121637ad0eefc8dbe76295d190bd8f41c90f7acc3c130510756d" Oct 30 22:34:11.668208 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:11.668173 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:13.911708 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:13.911677 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:16.568208 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:16.568173 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:21.365837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:21.365799 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:24.529728 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.529697 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7"] Oct 30 22:34:24.530123 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.530109 5508 cpu_manager.go:414] "containerMap: removing container" podUID="4d1079f7-e302-4965-86bd-ba7b8b7e81ca" containerName="revoke-keys" Oct 30 22:34:24.530165 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.530124 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1079f7-e302-4965-86bd-ba7b8b7e81ca" containerName="revoke-keys" Oct 30 22:34:24.530254 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.530240 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1079f7-e302-4965-86bd-ba7b8b7e81ca" containerName="revoke-keys" Oct 30 22:34:24.534897 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.534880 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.588645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.588619 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:24.637807 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.637774 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:24.637807 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.637775 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:24.638013 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.637787 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:34:24.639481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.639459 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7"] Oct 30 22:34:24.642497 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.642476 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88n9c\" (UniqueName: \"kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.642564 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.642512 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.642564 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.642539 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.743216 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.743185 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88n9c\" (UniqueName: \"kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.743216 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.743216 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.743482 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.743245 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.743947 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.743921 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.745768 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.745743 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.801001 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.800945 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88n9c\" (UniqueName: \"kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c\") pod \"maas-api-revoke-sub-7f8d21f1-5lwn7\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:24.848021 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:24.847997 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:25.219335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:25.219294 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7"] Oct 30 22:34:25.221819 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:34:25.221789 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409174d4_6987_485a_9b56_c30be1782694.slice/crio-c12f3cae9eb5efe02752d6dc4aa8a44e537a9c48b96149cb839c5db924c3225c WatchSource:0}: Error finding container c12f3cae9eb5efe02752d6dc4aa8a44e537a9c48b96149cb839c5db924c3225c: Status 404 returned error can't find the container with id c12f3cae9eb5efe02752d6dc4aa8a44e537a9c48b96149cb839c5db924c3225c Oct 30 22:34:25.469674 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:25.469590 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" event={"ID":"409174d4-6987-485a-9b56-c30be1782694","Type":"ContainerStarted","Data":"c12f3cae9eb5efe02752d6dc4aa8a44e537a9c48b96149cb839c5db924c3225c"} Oct 30 22:34:26.263605 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:26.263567 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:26.475716 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:26.475676 5508 generic.go:356] "Generic (PLEG): container finished" podID="409174d4-6987-485a-9b56-c30be1782694" containerID="2d0ddf3e5e2a37be049f8b56804a7ec48c6c513912b44cb5010df661b66a6305" exitCode=0 Oct 30 22:34:26.475889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:26.475759 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" event={"ID":"409174d4-6987-485a-9b56-c30be1782694","Type":"ContainerDied","Data":"2d0ddf3e5e2a37be049f8b56804a7ec48c6c513912b44cb5010df661b66a6305"} Oct 30 22:34:27.619292 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.619268 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:27.774423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.774292 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca\") pod \"409174d4-6987-485a-9b56-c30be1782694\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " Oct 30 22:34:27.774573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.774421 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca\") pod \"409174d4-6987-485a-9b56-c30be1782694\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " Oct 30 22:34:27.774573 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.774519 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c\" (UniqueName: \"kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c\") pod \"409174d4-6987-485a-9b56-c30be1782694\" (UID: \"409174d4-6987-485a-9b56-c30be1782694\") " Oct 30 22:34:27.774688 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.774626 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca" pod "409174d4-6987-485a-9b56-c30be1782694" (UID: "409174d4-6987-485a-9b56-c30be1782694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:27.774794 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.774778 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409174d4-6987-485a-9b56-c30be1782694-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:27.776606 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.776584 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca" pod "409174d4-6987-485a-9b56-c30be1782694" (UID: "409174d4-6987-485a-9b56-c30be1782694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:27.776856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.776834 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c" pod "409174d4-6987-485a-9b56-c30be1782694" (UID: "409174d4-6987-485a-9b56-c30be1782694"). InnerVolumeSpecName "kube-api-access-88n9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:27.876068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.876029 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88n9c\" (UniqueName: \"kubernetes.io/projected/409174d4-6987-485a-9b56-c30be1782694-kube-api-access-88n9c\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:27.876068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:27.876066 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409174d4-6987-485a-9b56-c30be1782694-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:28.485812 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:28.485780 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" Oct 30 22:34:28.486017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:28.485824 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7" event={"ID":"409174d4-6987-485a-9b56-c30be1782694","Type":"ContainerDied","Data":"c12f3cae9eb5efe02752d6dc4aa8a44e537a9c48b96149cb839c5db924c3225c"} Oct 30 22:34:28.486017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:28.485874 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c12f3cae9eb5efe02752d6dc4aa8a44e537a9c48b96149cb839c5db924c3225c" Oct 30 22:34:31.037458 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:31.037421 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:35.666401 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:35.666362 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:35.916079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:35.916012 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:36.172261 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.172226 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g"] Oct 30 22:34:36.172641 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.172628 5508 cpu_manager.go:414] "containerMap: removing container" podUID="409174d4-6987-485a-9b56-c30be1782694" containerName="revoke-keys" Oct 30 22:34:36.172692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.172643 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409174d4-6987-485a-9b56-c30be1782694" containerName="revoke-keys" Oct 30 22:34:36.172739 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.172730 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="409174d4-6987-485a-9b56-c30be1782694" containerName="revoke-keys" Oct 30 22:34:36.176181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.176154 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.227936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.227910 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:34:36.228114 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.227958 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:34:36.277268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.277242 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:34:36.277268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.277264 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:34:36.279063 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.279042 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g"] Oct 30 22:34:36.354661 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.354628 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.354804 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.354732 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.354804 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.354779 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98z5s\" (UniqueName: \"kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.455808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.455725 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.455808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.455785 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.455994 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.455817 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98z5s\" (UniqueName: \"kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.456471 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.456444 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.458265 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.458242 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.512127 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.512096 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98z5s\" (UniqueName: \"kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s\") pod \"maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:36.789130 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:36.789049 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:37.160897 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:37.160787 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g"] Oct 30 22:34:37.163987 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:34:37.163962 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8915fc99_d893_4717_919c_61a89be3afbd.slice/crio-f5b3d788e22bcb96780a83be479a12725d1a6ba1e209557235a505d6e49514f4 WatchSource:0}: Error finding container f5b3d788e22bcb96780a83be479a12725d1a6ba1e209557235a505d6e49514f4: Status 404 returned error can't find the container with id f5b3d788e22bcb96780a83be479a12725d1a6ba1e209557235a505d6e49514f4 Oct 30 22:34:37.530427 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:37.530393 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" event={"ID":"8915fc99-d893-4717-919c-61a89be3afbd","Type":"ContainerStarted","Data":"00677cd0c61ffdab99b97c9a6e28a077036fd10d41c887b8cefdf782d3c3625b"} Oct 30 22:34:37.530563 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:37.530450 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" event={"ID":"8915fc99-d893-4717-919c-61a89be3afbd","Type":"ContainerStarted","Data":"f5b3d788e22bcb96780a83be479a12725d1a6ba1e209557235a505d6e49514f4"} Oct 30 22:34:37.645807 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:37.645750 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" podStartSLOduration=1.6457356829999998 podStartE2EDuration="1.645735683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:37.644361327 +0000 UTC m=+3087.833899002" watchObservedRunningTime="2026-10-30 22:34:37.645735683 +0000 UTC m=+3087.835273357" Oct 30 22:34:38.538053 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:38.537958 5508 generic.go:356] "Generic (PLEG): container finished" podID="8915fc99-d893-4717-919c-61a89be3afbd" containerID="00677cd0c61ffdab99b97c9a6e28a077036fd10d41c887b8cefdf782d3c3625b" exitCode=0 Oct 30 22:34:38.538053 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:38.538013 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" event={"ID":"8915fc99-d893-4717-919c-61a89be3afbd","Type":"ContainerDied","Data":"00677cd0c61ffdab99b97c9a6e28a077036fd10d41c887b8cefdf782d3c3625b"} Oct 30 22:34:38.851335 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:38.851268 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:39.677786 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.677763 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:39.790982 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.790947 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca\") pod \"8915fc99-d893-4717-919c-61a89be3afbd\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " Oct 30 22:34:39.791196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.791020 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca\") pod \"8915fc99-d893-4717-919c-61a89be3afbd\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " Oct 30 22:34:39.791196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.791089 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s\" (UniqueName: \"kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s\") pod \"8915fc99-d893-4717-919c-61a89be3afbd\" (UID: \"8915fc99-d893-4717-919c-61a89be3afbd\") " Oct 30 22:34:39.791517 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.791486 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca" pod "8915fc99-d893-4717-919c-61a89be3afbd" (UID: "8915fc99-d893-4717-919c-61a89be3afbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:39.793894 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.793836 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca" pod "8915fc99-d893-4717-919c-61a89be3afbd" (UID: "8915fc99-d893-4717-919c-61a89be3afbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:39.794168 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.794141 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s" pod "8915fc99-d893-4717-919c-61a89be3afbd" (UID: "8915fc99-d893-4717-919c-61a89be3afbd"). InnerVolumeSpecName "kube-api-access-98z5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:39.892247 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.892212 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98z5s\" (UniqueName: \"kubernetes.io/projected/8915fc99-d893-4717-919c-61a89be3afbd-kube-api-access-98z5s\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:39.892247 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.892242 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:39.892443 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:39.892257 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8915fc99-d893-4717-919c-61a89be3afbd-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.371805 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:34:40.371757 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:34:40.549867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:40.549825 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" event={"ID":"8915fc99-d893-4717-919c-61a89be3afbd","Type":"ContainerDied","Data":"f5b3d788e22bcb96780a83be479a12725d1a6ba1e209557235a505d6e49514f4"} Oct 30 22:34:40.549867 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:40.549867 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g" Oct 30 22:34:40.550086 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:40.549886 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b3d788e22bcb96780a83be479a12725d1a6ba1e209557235a505d6e49514f4" Oct 30 22:34:40.676994 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:40.676911 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:41.289783 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:41.289754 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:46.180258 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:46.180222 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:48.044044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:48.044007 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:48.213902 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:48.213839 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:49.392089 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:49.392046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:50.975936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:50.975900 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:55.206219 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:55.206186 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:56.235004 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:56.234972 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:56.732511 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:56.732459 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:57.091044 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:57.091012 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:57.316033 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:57.316002 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g"] Oct 30 22:34:57.364725 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:57.364662 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-85b9fe-nmz6g"] Oct 30 22:34:57.757976 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:57.757891 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:57.951461 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:57.951428 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8915fc99-d893-4717-919c-61a89be3afbd" path="/var/lib/kubelet/pods/8915fc99-d893-4717-919c-61a89be3afbd/volumes" Oct 30 22:34:58.047038 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:58.046967 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85"] Oct 30 22:34:58.096096 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:58.096059 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a67cb3-tsf85"] Oct 30 22:34:59.954017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:34:59.953980 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaaa7b1f-3998-4ee7-b54d-8dc3d404303f" path="/var/lib/kubelet/pods/eaaa7b1f-3998-4ee7-b54d-8dc3d404303f/volumes" Oct 30 22:35:03.748158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:03.748121 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:03.901958 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:03.901922 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:04.091409 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:04.091370 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:10.372094 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:35:10.372052 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:35:11.224921 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:11.224887 5508 scope.go:122] "RemoveContainer" containerID="ff0cd47620ed4867710b2f933b3fae4b1883928d586022e2eca608f35dfdf691" Oct 30 22:35:14.000454 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:14.000421 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:17.566882 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:17.566846 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:18.355966 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:18.355926 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:23.268459 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:23.268411 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:35:23.268964 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:23.268479 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vmg92" podUID="889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:35:38.047640 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:38.047603 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g"] Oct 30 22:35:38.096801 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:38.096757 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t626g"] Oct 30 22:35:39.136820 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:39.136782 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:39.952031 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:39.952000 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd03ccbc-a80e-4751-9e43-2b07b890bba3" path="/var/lib/kubelet/pods/cd03ccbc-a80e-4751-9e43-2b07b890bba3/volumes" Oct 30 22:35:40.372551 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:35:40.372515 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:35:41.170097 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:41.170064 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:57.976079 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:35:57.976003 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:10.372050 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:36:10.372012 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:36:11.246025 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:11.245992 5508 scope.go:122] "RemoveContainer" containerID="a2cdcb8914603d99081f1308db122177fb89c6279ffd93d1cf76c919e87cae42" Oct 30 22:36:11.254675 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:11.254651 5508 scope.go:122] "RemoveContainer" containerID="9d93b007830fe7640582d4bc3f17e4d2bbefbeadc1678b6b7be534668b57dc2e" Oct 30 22:36:27.722279 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.722244 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt"] Oct 30 22:36:27.724530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.722698 5508 cpu_manager.go:414] "containerMap: removing container" podUID="8915fc99-d893-4717-919c-61a89be3afbd" containerName="revoke-keys" Oct 30 22:36:27.724530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.722713 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8915fc99-d893-4717-919c-61a89be3afbd" containerName="revoke-keys" Oct 30 22:36:27.724530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.722793 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8915fc99-d893-4717-919c-61a89be3afbd" containerName="revoke-keys" Oct 30 22:36:27.725569 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.725554 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.826537 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.826501 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:27.826537 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.826507 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:27.826814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.826594 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:27.826814 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.826725 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:36:27.828280 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.828260 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt"] Oct 30 22:36:27.840885 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.840861 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.841029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.840906 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.841029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.840941 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79dx\" (UniqueName: \"kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.942054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.942021 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.942054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.942074 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.942397 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.942112 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79dx\" (UniqueName: \"kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.942923 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.942828 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.944691 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.944664 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:27.999037 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:27.998971 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79dx\" (UniqueName: \"kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx\") pod \"maas-api-revoke-sub-452b6d7b-n5bjt\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:28.038494 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:28.038466 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:28.409688 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:28.409661 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt"] Oct 30 22:36:28.412688 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:36:28.412662 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840a767f_26fd_4510_a3be_c873a801e35c.slice/crio-86d6dcd46f717efcf24b8d8ce0f75ce4ba7c54055bcc7ca2e552902a0f98aaeb WatchSource:0}: Error finding container 86d6dcd46f717efcf24b8d8ce0f75ce4ba7c54055bcc7ca2e552902a0f98aaeb: Status 404 returned error can't find the container with id 86d6dcd46f717efcf24b8d8ce0f75ce4ba7c54055bcc7ca2e552902a0f98aaeb Oct 30 22:36:29.082617 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:29.082587 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" event={"ID":"840a767f-26fd-4510-a3be-c873a801e35c","Type":"ContainerStarted","Data":"6894c3c01fd7620043f35ef3548a9deae04e2c99249d32000b14aefe4001df9c"} Oct 30 22:36:29.083081 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:29.082642 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" event={"ID":"840a767f-26fd-4510-a3be-c873a801e35c","Type":"ContainerStarted","Data":"86d6dcd46f717efcf24b8d8ce0f75ce4ba7c54055bcc7ca2e552902a0f98aaeb"} Oct 30 22:36:29.196815 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:29.196756 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" podStartSLOduration=2.196741645 podStartE2EDuration="2.196741645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:29.196019589 +0000 UTC m=+3199.385557263" watchObservedRunningTime="2026-10-30 22:36:29.196741645 +0000 UTC m=+3199.386279321" Oct 30 22:36:30.090279 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:30.090234 5508 generic.go:356] "Generic (PLEG): container finished" podID="840a767f-26fd-4510-a3be-c873a801e35c" containerID="6894c3c01fd7620043f35ef3548a9deae04e2c99249d32000b14aefe4001df9c" exitCode=0 Oct 30 22:36:30.090832 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:30.090337 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" event={"ID":"840a767f-26fd-4510-a3be-c873a801e35c","Type":"ContainerDied","Data":"6894c3c01fd7620043f35ef3548a9deae04e2c99249d32000b14aefe4001df9c"} Oct 30 22:36:31.502824 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.502478 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:31.589622 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.589588 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx\" (UniqueName: \"kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx\") pod \"840a767f-26fd-4510-a3be-c873a801e35c\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " Oct 30 22:36:31.590391 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.590364 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca\") pod \"840a767f-26fd-4510-a3be-c873a801e35c\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " Oct 30 22:36:31.590994 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.590973 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca\") pod \"840a767f-26fd-4510-a3be-c873a801e35c\" (UID: \"840a767f-26fd-4510-a3be-c873a801e35c\") " Oct 30 22:36:31.591727 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.590845 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca" pod "840a767f-26fd-4510-a3be-c873a801e35c" (UID: "840a767f-26fd-4510-a3be-c873a801e35c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:31.633134 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.624697 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca" pod "840a767f-26fd-4510-a3be-c873a801e35c" (UID: "840a767f-26fd-4510-a3be-c873a801e35c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:31.633238 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.631937 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx" pod "840a767f-26fd-4510-a3be-c873a801e35c" (UID: "840a767f-26fd-4510-a3be-c873a801e35c"). InnerVolumeSpecName "kube-api-access-t79dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.714421 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.692987 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.714707 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.714666 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79dx\" (UniqueName: \"kubernetes.io/projected/840a767f-26fd-4510-a3be-c873a801e35c-kube-api-access-t79dx\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.714882 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:31.714865 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/840a767f-26fd-4510-a3be-c873a801e35c-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.143733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:32.143692 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" event={"ID":"840a767f-26fd-4510-a3be-c873a801e35c","Type":"ContainerDied","Data":"86d6dcd46f717efcf24b8d8ce0f75ce4ba7c54055bcc7ca2e552902a0f98aaeb"} Oct 30 22:36:32.144052 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:32.144011 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d6dcd46f717efcf24b8d8ce0f75ce4ba7c54055bcc7ca2e552902a0f98aaeb" Oct 30 22:36:32.144484 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:32.144453 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt" Oct 30 22:36:40.372284 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:36:40.372245 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:36:41.823116 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:41.823073 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:41.976128 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:41.976095 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:45.161295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:45.161250 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:49.157630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.157594 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8"] Oct 30 22:36:49.158012 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.158004 5508 cpu_manager.go:414] "containerMap: removing container" podUID="840a767f-26fd-4510-a3be-c873a801e35c" containerName="revoke-keys" Oct 30 22:36:49.158054 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.158017 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840a767f-26fd-4510-a3be-c873a801e35c" containerName="revoke-keys" Oct 30 22:36:49.158114 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.158102 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="840a767f-26fd-4510-a3be-c873a801e35c" containerName="revoke-keys" Oct 30 22:36:49.161357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.161336 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.241328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.241264 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.241502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.241440 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltqz\" (UniqueName: \"kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.241502 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.241484 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.263441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.263413 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:36:49.263771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.263632 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:36:49.263771 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.263762 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:36:49.263950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.263722 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:36:49.264658 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.264640 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8"] Oct 30 22:36:49.342475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.342449 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.342615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.342501 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltqz\" (UniqueName: \"kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.342615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.342542 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.343196 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.343178 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.344733 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.344717 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.398551 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.398525 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltqz\" (UniqueName: \"kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz\") pod \"maas-api-revoke-sub-3e31a506-825g8\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.474820 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.474762 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:49.846010 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:49.845893 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8"] Oct 30 22:36:49.848871 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:36:49.848845 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dad3485_8853_4dcc_8a8e_540a316b7bdf.slice/crio-080bcf790788ce1ec4ec7ce154e0240b136581527aed3f4647411e8d25592a1f WatchSource:0}: Error finding container 080bcf790788ce1ec4ec7ce154e0240b136581527aed3f4647411e8d25592a1f: Status 404 returned error can't find the container with id 080bcf790788ce1ec4ec7ce154e0240b136581527aed3f4647411e8d25592a1f Oct 30 22:36:50.259051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:50.259020 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" event={"ID":"9dad3485-8853-4dcc-8a8e-540a316b7bdf","Type":"ContainerStarted","Data":"aa1b75787e8c612baec2522798949193314e28307c43586e3f014b82c2170a98"} Oct 30 22:36:50.259439 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:50.259072 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" event={"ID":"9dad3485-8853-4dcc-8a8e-540a316b7bdf","Type":"ContainerStarted","Data":"080bcf790788ce1ec4ec7ce154e0240b136581527aed3f4647411e8d25592a1f"} Oct 30 22:36:50.372021 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:50.371959 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" podStartSLOduration=1.37194541 podStartE2EDuration="1.37194541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:50.370994292 +0000 UTC m=+3220.560531969" watchObservedRunningTime="2026-10-30 22:36:50.37194541 +0000 UTC m=+3220.561483082" Oct 30 22:36:51.266295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:51.266256 5508 generic.go:356] "Generic (PLEG): container finished" podID="9dad3485-8853-4dcc-8a8e-540a316b7bdf" containerID="aa1b75787e8c612baec2522798949193314e28307c43586e3f014b82c2170a98" exitCode=0 Oct 30 22:36:51.266698 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:51.266326 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" event={"ID":"9dad3485-8853-4dcc-8a8e-540a316b7bdf","Type":"ContainerDied","Data":"aa1b75787e8c612baec2522798949193314e28307c43586e3f014b82c2170a98"} Oct 30 22:36:52.598328 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.588465 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:52.695796 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.695750 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca\") pod \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " Oct 30 22:36:52.696989 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.696592 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz\" (UniqueName: \"kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz\") pod \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " Oct 30 22:36:52.697256 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.696509 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca" pod "9dad3485-8853-4dcc-8a8e-540a316b7bdf" (UID: "9dad3485-8853-4dcc-8a8e-540a316b7bdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:52.697498 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.697479 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca\") pod \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\" (UID: \"9dad3485-8853-4dcc-8a8e-540a316b7bdf\") " Oct 30 22:36:52.698646 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.698627 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:52.720467 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.720418 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca" pod "9dad3485-8853-4dcc-8a8e-540a316b7bdf" (UID: "9dad3485-8853-4dcc-8a8e-540a316b7bdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:52.720620 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.720467 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz" pod "9dad3485-8853-4dcc-8a8e-540a316b7bdf" (UID: "9dad3485-8853-4dcc-8a8e-540a316b7bdf"). InnerVolumeSpecName "kube-api-access-5ltqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:52.810388 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.810086 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ltqz\" (UniqueName: \"kubernetes.io/projected/9dad3485-8853-4dcc-8a8e-540a316b7bdf-kube-api-access-5ltqz\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:52.810388 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:52.810123 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad3485-8853-4dcc-8a8e-540a316b7bdf-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:36:53.303331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:53.299104 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" event={"ID":"9dad3485-8853-4dcc-8a8e-540a316b7bdf","Type":"ContainerDied","Data":"080bcf790788ce1ec4ec7ce154e0240b136581527aed3f4647411e8d25592a1f"} Oct 30 22:36:53.303331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:53.299175 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080bcf790788ce1ec4ec7ce154e0240b136581527aed3f4647411e8d25592a1f" Oct 30 22:36:53.303331 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:53.299253 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8" Oct 30 22:36:54.679614 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:54.679582 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:54.840628 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:36:54.840588 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:10.385690 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:37:10.385597 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:37:10.731523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:10.731402 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:37:10.731816 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:10.731767 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:37:11.280688 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:11.280654 5508 scope.go:122] "RemoveContainer" containerID="d851e12509d91236c5c2394c95d3c888753e8ee8d4146ff946c926d9b8fd6875" Oct 30 22:37:18.657076 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.656849 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp"] Oct 30 22:37:18.657630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.657384 5508 cpu_manager.go:414] "containerMap: removing container" podUID="9dad3485-8853-4dcc-8a8e-540a316b7bdf" containerName="revoke-keys" Oct 30 22:37:18.657630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.657404 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dad3485-8853-4dcc-8a8e-540a316b7bdf" containerName="revoke-keys" Oct 30 22:37:18.657630 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.657530 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dad3485-8853-4dcc-8a8e-540a316b7bdf" containerName="revoke-keys" Oct 30 22:37:18.662489 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.662465 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.727092 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.727049 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.727267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.727121 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9psn6\" (UniqueName: \"kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.727456 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.727328 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.764706 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.764678 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:37:18.764706 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.764684 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:18.764905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.764768 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:18.764905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.764824 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:37:18.766700 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.766676 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp"] Oct 30 22:37:18.827872 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.827841 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9psn6\" (UniqueName: \"kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.828020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.827929 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.828020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.827962 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.828655 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.828628 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.830598 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.830576 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.888543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.888523 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9psn6\" (UniqueName: \"kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6\") pod \"maas-api-revoke-sub-d9fa49d6-h47hp\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:18.975277 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:18.975207 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:19.348615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:19.348587 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp"] Oct 30 22:37:19.350231 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:37:19.350204 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c7e943_5f43_4a29_8783_944a70c44e70.slice/crio-a9c95896426d7c5a58ad66fae9ddd7ceaf2e175dfa4995448344b4101105ab08 WatchSource:0}: Error finding container a9c95896426d7c5a58ad66fae9ddd7ceaf2e175dfa4995448344b4101105ab08: Status 404 returned error can't find the container with id a9c95896426d7c5a58ad66fae9ddd7ceaf2e175dfa4995448344b4101105ab08 Oct 30 22:37:19.447052 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:19.447023 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" event={"ID":"26c7e943-5f43-4a29-8783-944a70c44e70","Type":"ContainerStarted","Data":"a9c95896426d7c5a58ad66fae9ddd7ceaf2e175dfa4995448344b4101105ab08"} Oct 30 22:37:20.453753 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:20.453719 5508 generic.go:356] "Generic (PLEG): container finished" podID="26c7e943-5f43-4a29-8783-944a70c44e70" containerID="65e9bb198eb5d377506febf2dec8c9b9b5b82f848fa1d7fa417e288a79acc3f5" exitCode=0 Oct 30 22:37:20.454122 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:20.453766 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" event={"ID":"26c7e943-5f43-4a29-8783-944a70c44e70","Type":"ContainerDied","Data":"65e9bb198eb5d377506febf2dec8c9b9b5b82f848fa1d7fa417e288a79acc3f5"} Oct 30 22:37:21.595637 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.595609 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:21.653481 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.653459 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca\") pod \"26c7e943-5f43-4a29-8783-944a70c44e70\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " Oct 30 22:37:21.653591 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.653498 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca\") pod \"26c7e943-5f43-4a29-8783-944a70c44e70\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " Oct 30 22:37:21.653639 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.653594 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6\" (UniqueName: \"kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6\") pod \"26c7e943-5f43-4a29-8783-944a70c44e70\" (UID: \"26c7e943-5f43-4a29-8783-944a70c44e70\") " Oct 30 22:37:21.653780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.653759 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca" pod "26c7e943-5f43-4a29-8783-944a70c44e70" (UID: "26c7e943-5f43-4a29-8783-944a70c44e70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:21.653889 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.653876 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-openshift-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.655745 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.655723 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca" pod "26c7e943-5f43-4a29-8783-944a70c44e70" (UID: "26c7e943-5f43-4a29-8783-944a70c44e70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:21.655860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.655790 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6" pod "26c7e943-5f43-4a29-8783-944a70c44e70" (UID: "26c7e943-5f43-4a29-8783-944a70c44e70"). InnerVolumeSpecName "kube-api-access-9psn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:21.754975 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.754897 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9psn6\" (UniqueName: \"kubernetes.io/projected/26c7e943-5f43-4a29-8783-944a70c44e70-kube-api-access-9psn6\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:21.754975 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:21.754929 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c7e943-5f43-4a29-8783-944a70c44e70-maas-api-xks-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:22.464259 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:22.464226 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" Oct 30 22:37:22.464446 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:22.464227 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp" event={"ID":"26c7e943-5f43-4a29-8783-944a70c44e70","Type":"ContainerDied","Data":"a9c95896426d7c5a58ad66fae9ddd7ceaf2e175dfa4995448344b4101105ab08"} Oct 30 22:37:22.464446 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:22.464366 5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c95896426d7c5a58ad66fae9ddd7ceaf2e175dfa4995448344b4101105ab08" Oct 30 22:37:29.047457 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:29.047420 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt"] Oct 30 22:37:29.096740 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:29.096702 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7c7kt"] Oct 30 22:37:29.953752 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:29.953713 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeed1c66-a1ed-44c0-b621-244906ea15c8" path="/var/lib/kubelet/pods/aeed1c66-a1ed-44c0-b621-244906ea15c8/volumes" Oct 30 22:37:31.878384 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.878344 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj"] Oct 30 22:37:31.878981 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.878965 5508 cpu_manager.go:414] "containerMap: removing container" podUID="26c7e943-5f43-4a29-8783-944a70c44e70" containerName="revoke-keys" Oct 30 22:37:31.879029 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.878983 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c7e943-5f43-4a29-8783-944a70c44e70" containerName="revoke-keys" Oct 30 22:37:31.879105 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.879094 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c7e943-5f43-4a29-8783-944a70c44e70" containerName="revoke-keys" Oct 30 22:37:31.883737 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.883718 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:31.955456 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.955404 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:31.955665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.955494 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xcc\" (UniqueName: \"kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:31.955665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.955547 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:31.955665 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.955601 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:31.988325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.986954 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:37:31.988325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.987037 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:37:31.988325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.987866 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pjcd6\"" Oct 30 22:37:31.988325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.987990 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:37:31.988325 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.988058 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:37:31.989748 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:31.989711 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj"] Oct 30 22:37:32.056963 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.056925 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.057172 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.057004 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.057172 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.057062 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xcc\" (UniqueName: \"kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.057172 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.057122 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.057694 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.057671 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.059644 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.059622 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.059808 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.059787 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.112773 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.112745 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xcc\" (UniqueName: \"kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc\") pod \"maas-api-79787b59b7-cvqfj\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.197128 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.197054 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:32.571343 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.571291 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj"] Oct 30 22:37:32.575058 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:37:32.575031 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3c617c_e6b1_45e7_b8ab_3febc2d2dcaf.slice/crio-317ddd527d7987eba8b24ed0825d7ca2cd18e56d33f92b8958884f86b7126b47 WatchSource:0}: Error finding container 317ddd527d7987eba8b24ed0825d7ca2cd18e56d33f92b8958884f86b7126b47: Status 404 returned error can't find the container with id 317ddd527d7987eba8b24ed0825d7ca2cd18e56d33f92b8958884f86b7126b47 Oct 30 22:37:32.576744 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:32.576725 5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:37:33.518444 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:33.518403 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" event={"ID":"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf","Type":"ContainerStarted","Data":"03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299"} Oct 30 22:37:33.518785 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:33.518462 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" event={"ID":"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf","Type":"ContainerStarted","Data":"317ddd527d7987eba8b24ed0825d7ca2cd18e56d33f92b8958884f86b7126b47"} Oct 30 22:37:33.518785 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:33.518544 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:33.654445 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:33.654350 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" podStartSLOduration=1.8643340510000002 podStartE2EDuration="2.654329844s" totalImagesPullingTime="789.995793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:33.652693578 +0000 UTC m=+3263.842231264" watchObservedRunningTime="2026-10-30 22:37:33.654329844 +0000 UTC m=+3263.843867515" Oct 30 22:37:38.394955 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:38.394913 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:38.991803 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:38.991764 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:39.259297 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:39.259205 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:39.528367 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:39.528270 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:39.817632 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:39.817543 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:40.371554 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:37:40.371519 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:37:40.797529 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.797450 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64979f7667-fb9sf"] Oct 30 22:37:40.800746 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.800730 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:40.903519 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.903491 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64979f7667-fb9sf"] Oct 30 22:37:40.950334 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.950279 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zd42\" (UniqueName: \"kubernetes.io/projected/9aa978b8-650c-4074-beb0-5981a3cab5d0-kube-api-access-8zd42\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:40.950507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.950468 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa978b8-650c-4074-beb0-5981a3cab5d0-maas-api-tls\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:40.950507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.950495 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa978b8-650c-4074-beb0-5981a3cab5d0-metrics-certs\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:40.950644 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:40.950523 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa978b8-650c-4074-beb0-5981a3cab5d0-openshift-service-ca\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.051829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.051752 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa978b8-650c-4074-beb0-5981a3cab5d0-maas-api-tls\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.051829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.051788 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa978b8-650c-4074-beb0-5981a3cab5d0-metrics-certs\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.051829 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.051818 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa978b8-650c-4074-beb0-5981a3cab5d0-openshift-service-ca\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.052113 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.051848 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zd42\" (UniqueName: \"kubernetes.io/projected/9aa978b8-650c-4074-beb0-5981a3cab5d0-kube-api-access-8zd42\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.052609 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.052581 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aa978b8-650c-4074-beb0-5981a3cab5d0-openshift-service-ca\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.054668 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.054640 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9aa978b8-650c-4074-beb0-5981a3cab5d0-maas-api-tls\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.054777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.054706 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9aa978b8-650c-4074-beb0-5981a3cab5d0-metrics-certs\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.107887 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.107864 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zd42\" (UniqueName: \"kubernetes.io/projected/9aa978b8-650c-4074-beb0-5981a3cab5d0-kube-api-access-8zd42\") pod \"maas-api-64979f7667-fb9sf\" (UID: \"9aa978b8-650c-4074-beb0-5981a3cab5d0\") " pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.113696 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.113668 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:41.487680 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.487646 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64979f7667-fb9sf"] Oct 30 22:37:41.491282 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:37:41.491255 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa978b8_650c_4074_beb0_5981a3cab5d0.slice/crio-4348d032162af2108c52569ac6eaf07cbd117396450d065701694dffec10d406 WatchSource:0}: Error finding container 4348d032162af2108c52569ac6eaf07cbd117396450d065701694dffec10d406: Status 404 returned error can't find the container with id 4348d032162af2108c52569ac6eaf07cbd117396450d065701694dffec10d406 Oct 30 22:37:41.565423 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:41.565393 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" event={"ID":"9aa978b8-650c-4074-beb0-5981a3cab5d0","Type":"ContainerStarted","Data":"4348d032162af2108c52569ac6eaf07cbd117396450d065701694dffec10d406"} Oct 30 22:37:42.571965 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:42.571928 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" event={"ID":"9aa978b8-650c-4074-beb0-5981a3cab5d0","Type":"ContainerStarted","Data":"643ec54c7fefb4a486e19c2477c8b9940cf65c287d6819f38737b2b00d5b6a94"} Oct 30 22:37:42.572470 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:42.572060 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:42.687925 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:42.687834 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" podStartSLOduration=1.847589734 podStartE2EDuration="2.687821367s" totalImagesPullingTime="840.231633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:42.686079881 +0000 UTC m=+3272.875617570" watchObservedRunningTime="2026-10-30 22:37:42.687821367 +0000 UTC m=+3272.877359042" Oct 30 22:37:42.971379 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:42.971290 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:43.188363 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:43.188330 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:43.354876 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:43.354843 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:46.207633 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:46.207599 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:46.382835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:46.382793 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:47.274747 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:47.274715 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:47.425926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:47.425892 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:47.584556 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:47.584506 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:48.582834 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:48.582804 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64979f7667-fb9sf" Oct 30 22:37:48.748957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:48.748925 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj"] Oct 30 22:37:48.749450 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:48.749411 5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" podUID="7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" containerName="maas-api" containerID="cri-o://03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299" gracePeriod=30 Oct 30 22:37:49.003709 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.003678 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:49.033950 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.033922 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca\") pod \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " Oct 30 22:37:49.034068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.033977 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs\") pod \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " Oct 30 22:37:49.034068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.034007 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls\") pod \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " Oct 30 22:37:49.034068 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.034036 5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc\" (UniqueName: \"kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc\") pod \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\" (UID: \"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf\") " Oct 30 22:37:49.034285 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.034257 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca" pod "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" (UID: "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:49.034506 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.034481 5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-openshift-service-ca\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.036187 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.036164 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs" pod "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" (UID: "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:49.036275 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.036257 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls" pod "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" (UID: "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:49.036362 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.036339 5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc" pod "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" (UID: "7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf"). InnerVolumeSpecName "kube-api-access-q5xcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:49.135268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.135203 5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-metrics-certs\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.135268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.135232 5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-maas-api-tls\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.135268 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.135243 5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5xcc\" (UniqueName: \"kubernetes.io/projected/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf-kube-api-access-q5xcc\") on node \"ip-10-0-134-73.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:37:49.607332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.607274 5508 generic.go:356] "Generic (PLEG): container finished" podID="7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" containerID="03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299" exitCode=0 Oct 30 22:37:49.607825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.607362 5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" Oct 30 22:37:49.607825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.607361 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" event={"ID":"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf","Type":"ContainerDied","Data":"03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299"} Oct 30 22:37:49.607825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.607490 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj" event={"ID":"7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf","Type":"ContainerDied","Data":"317ddd527d7987eba8b24ed0825d7ca2cd18e56d33f92b8958884f86b7126b47"} Oct 30 22:37:49.607825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.607513 5508 scope.go:122] "RemoveContainer" containerID="03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299" Oct 30 22:37:49.617167 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.617148 5508 scope.go:122] "RemoveContainer" containerID="03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299" Oct 30 22:37:49.617484 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:37:49.617465 5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299\": container with ID starting with 03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299 not found: ID does not exist" containerID="03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299" Oct 30 22:37:49.617543 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.617492 5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299"} err="failed to get container status \"03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299\": rpc error: code = NotFound desc = could not find container \"03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299\": container with ID starting with 03db6d02747b613ad55eaf7dbb87ffdbfff8964debc02a465dc21b8618cd0299 not found: ID does not exist" Oct 30 22:37:49.771157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.771118 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj"] Oct 30 22:37:49.821530 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.821394 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-79787b59b7-cvqfj"] Oct 30 22:37:49.951948 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:49.951878 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" path="/var/lib/kubelet/pods/7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf/volumes" Oct 30 22:37:52.468181 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:52.468146 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:57.428701 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:57.428665 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:58.045559 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:58.045526 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd"] Oct 30 22:37:58.094692 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:58.094666 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-kkkcd"] Oct 30 22:37:59.951863 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:37:59.951829 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1079f7-e302-4965-86bd-ba7b8b7e81ca" path="/var/lib/kubelet/pods/4d1079f7-e302-4965-86bd-ba7b8b7e81ca/volumes" Oct 30 22:38:02.277851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:02.277790 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:07.529851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:07.529814 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:10.376735 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:38:10.376691 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:38:12.706675 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:12.706611 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:17.677101 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:17.677061 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:22.485749 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:22.485711 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:27.453507 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:27.453471 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:31.205167 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:31.205131 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:31.355460 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:31.355424 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:32.259799 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:32.259759 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:36.533353 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:36.533295 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:38.004871 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:38.004806 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:38.275228 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:38.275143 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:40.162035 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:40.161997 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:40.318651 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:40.318616 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:40.372659 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:38:40.372621 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:38:41.513056 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:41.513024 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:42.209089 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:42.201287 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:42.363746 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:42.363708 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:45.805441 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:45.805396 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:46.081184 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:46.081157 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:49.022476 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:49.022392 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:49.309438 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:49.309338 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:49.488477 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:49.488441 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:53.113050 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:53.113007 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:53.259886 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:53.259854 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:53.416761 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:53.416677 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:38:58.081772 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:38:58.081742 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:03.009452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:03.009421 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:07.868957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:07.868928 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:10.372355 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:39:10.372290 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:39:10.584895 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:10.584861 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:11.313660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:11.313623 5508 scope.go:122] "RemoveContainer" containerID="b5d4416d8911be8b95092559e08fc5eecad7a44d8ba1d8e5af862a542acac7ec" Oct 30 22:39:11.324270 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:11.324244 5508 scope.go:122] "RemoveContainer" containerID="f53f50728d2c8ab6434f1bfce9f70c22e06c8e62d52977888950b95bcd9577ef" Oct 30 22:39:13.091434 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:13.091395 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:17.967083 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:17.967046 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:22.869345 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:22.869281 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:27.668017 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:27.667984 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:28.049157 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:28.049072 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7"] Oct 30 22:39:28.092452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:28.092422 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8d21f1-5lwn7"] Oct 30 22:39:29.953793 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:29.953756 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409174d4-6987-485a-9b56-c30be1782694" path="/var/lib/kubelet/pods/409174d4-6987-485a-9b56-c30be1782694/volumes" Oct 30 22:39:32.569158 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:32.569123 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:37.477835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:37.477771 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:40.372690 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:39:40.372654 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:39:40.512020 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:40.511951 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:42.368783 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:42.368741 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:47.208821 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:47.208781 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:52.182295 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:52.182255 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:56.975860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:56.975821 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:39:59.337960 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:39:59.337923 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:01.870164 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:01.870123 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:06.672371 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:06.672326 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:10.372636 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:40:10.372595 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:40:11.575373 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:11.575326 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:16.475926 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:16.475878 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:21.355534 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:21.355483 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:26.160548 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:26.160480 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:31.192130 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:31.190926 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:36.419483 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:36.419442 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:40.373408 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:40:40.373364 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:40:40.836367 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:40.836329 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:41.268760 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:41.268679 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:42.826221 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:42.826181 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:43.025838 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:43.025803 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:46.099347 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:46.099281 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:55.401369 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:55.401332 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:56.944663 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:56.944632 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:57.206946 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:57.206860 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:40:57.956121 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:40:57.956083 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:41:03.807780 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:03.807745 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:41:03.981413 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:03.981369 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:41:04.186485 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:04.186400 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:41:10.431090 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:41:10.431046 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:41:11.356138 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:11.356105 5508 scope.go:122] "RemoveContainer" containerID="00677cd0c61ffdab99b97c9a6e28a077036fd10d41c887b8cefdf782d3c3625b" Oct 30 22:41:11.364835 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:11.364811 5508 scope.go:122] "RemoveContainer" containerID="2d0ddf3e5e2a37be049f8b56804a7ec48c6c513912b44cb5010df661b66a6305" Oct 30 22:41:25.720330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:25.719449 5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:41:25.720330 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:25.719525 5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs" podUID="b5cf4322-153f-4647-b6f2-3e08ca5285f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:41:32.046660 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:32.046620 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt"] Oct 30 22:41:32.095927 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:32.095895 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-n5bjt"] Oct 30 22:41:33.951288 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:33.951258 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840a767f-26fd-4510-a3be-c873a801e35c" path="/var/lib/kubelet/pods/840a767f-26fd-4510-a3be-c873a801e35c/volumes" Oct 30 22:41:40.373287 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:41:40.373252 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:41:41.197354 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:41.197297 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:41:53.043281 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:53.043201 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8"] Oct 30 22:41:53.092453 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:53.092423 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-825g8"] Oct 30 22:41:53.950996 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:41:53.950963 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dad3485-8853-4dcc-8a8e-540a316b7bdf" path="/var/lib/kubelet/pods/9dad3485-8853-4dcc-8a8e-540a316b7bdf/volumes" Oct 30 22:42:10.371731 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:42:10.371692 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:42:22.047938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:22.047902 5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp"] Oct 30 22:42:22.096804 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:22.096767 5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9fa49d6-h47hp"] Oct 30 22:42:23.773775 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:23.773739 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:42:23.926475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:23.926426 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:42:23.952161 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:23.952121 5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c7e943-5f43-4a29-8783-944a70c44e70" path="/var/lib/kubelet/pods/26c7e943-5f43-4a29-8783-944a70c44e70/volumes" Oct 30 22:42:24.660822 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:24.660782 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b859fd7fd-z5xbg_a44828d8-ee41-4682-93f9-726b3560a5f6/manager/0.log" Oct 30 22:42:25.776218 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:25.776183 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:42:26.085559 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:26.085523 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:42:27.925615 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:27.925588 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64979f7667-fb9sf_9aa978b8-650c-4074-beb0-5981a3cab5d0/maas-api/0.log" Oct 30 22:42:31.379522 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:31.379494 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m84rz_f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99/kuadrant-console-plugin/0.log" Oct 30 22:42:33.148357 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:33.148330 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r8zrg_b88dbae2-8521-4946-a12b-7d44e60b04f1/discovery/0.log" Oct 30 22:42:33.361922 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:33.361892 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cbcf87c76-7h2zh_d755f93c-fd87-4cb6-8bd0-5c331e10d8eb/kube-auth-proxy/0.log" Oct 30 22:42:34.032094 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:34.032066 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs_b5cf4322-153f-4647-b6f2-3e08ca5285f5/istio-proxy/0.log" Oct 30 22:42:35.518249 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:35.518217 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k_ead1be97-52c5-489a-b659-f9342332885a/istio-proxy/0.log" Oct 30 22:42:35.743452 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:35.743417 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-vmg92_889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1/istio-proxy/0.log" Oct 30 22:42:38.299267 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:38.299240 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zr9gl_26ef4e87-d54c-4200-b643-3f1bcdce306c/main/0.log" Oct 30 22:42:39.067332 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:39.067289 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld_81e8a72c-35ee-4b3d-8466-6aa5adec13ab/main/0.log" Oct 30 22:42:39.540128 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:39.540098 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-g2572_7663bb39-fca3-4911-9af6-fc39c5ffa391/main/0.log" Oct 30 22:42:39.787904 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:39.787873 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r_b1b6787b-0e4c-4564-a9f0-5e84ea279f38/main/0.log" Oct 30 22:42:40.372405 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:42:40.372370 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:42:41.544111 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:41.544080 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:42:49.716984 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:49.716951 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6b859fd7fd-z5xbg_a44828d8-ee41-4682-93f9-726b3560a5f6/manager/0.log" Oct 30 22:42:52.381446 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:52.381417 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64979f7667-fb9sf_9aa978b8-650c-4074-beb0-5981a3cab5d0/maas-api/0.log" Oct 30 22:42:55.516252 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:55.516222 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m84rz_f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99/kuadrant-console-plugin/0.log" Oct 30 22:42:57.040395 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:57.040366 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r8zrg_b88dbae2-8521-4946-a12b-7d44e60b04f1/discovery/0.log" Oct 30 22:42:57.265348 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:57.265290 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cbcf87c76-7h2zh_d755f93c-fd87-4cb6-8bd0-5c331e10d8eb/kube-auth-proxy/0.log" Oct 30 22:42:57.895180 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:57.895151 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4jlxs_b5cf4322-153f-4647-b6f2-3e08ca5285f5/istio-proxy/0.log" Oct 30 22:42:59.334856 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:59.334821 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-vdw5k_ead1be97-52c5-489a-b659-f9342332885a/istio-proxy/0.log" Oct 30 22:42:59.546634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:42:59.546606 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-vmg92_889f4f10-f0e9-4c8e-95ff-c2f02d1c04f1/istio-proxy/0.log" Oct 30 22:43:01.467645 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:01.467613 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zr9gl_26ef4e87-d54c-4200-b643-3f1bcdce306c/main/0.log" Oct 30 22:43:02.218905 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:02.218867 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8275ld_81e8a72c-35ee-4b3d-8466-6aa5adec13ab/main/0.log" Oct 30 22:43:02.742976 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:02.742950 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-g2572_7663bb39-fca3-4911-9af6-fc39c5ffa391/main/0.log" Oct 30 22:43:03.011193 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:03.011106 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4gk2r_b1b6787b-0e4c-4564-a9f0-5e84ea279f38/main/0.log" Oct 30 22:43:10.373162 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:43:10.373124 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:43:11.445279 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:11.445248 5508 scope.go:122] "RemoveContainer" containerID="aa1b75787e8c612baec2522798949193314e28307c43586e3f014b82c2170a98" Oct 30 22:43:11.454032 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:11.454008 5508 scope.go:122] "RemoveContainer" containerID="6894c3c01fd7620043f35ef3548a9deae04e2c99249d32000b14aefe4001df9c" Oct 30 22:43:40.372038 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:43:40.372001 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:43:41.898704 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:43:41.898671 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:44:10.372232 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:44:10.372198 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:44:11.487027 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:11.486991 5508 scope.go:122] "RemoveContainer" containerID="65e9bb198eb5d377506febf2dec8c9b9b5b82f848fa1d7fa417e288a79acc3f5" Oct 30 22:44:12.028088 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:12.028053 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:44:12.218269 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:12.218232 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:44:21.287634 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:21.287594 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-56vhn_04ecd634-c544-41e3-82bd-b5e36a5d57cd/global-pull-secret-syncer/0.log" Oct 30 22:44:24.393987 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:24.393945 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" Oct 30 22:44:26.153201 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:26.153173 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-73.us-west-1.compute.internal_0994e35dd707326725d576ce662475d4/haproxy/0.log" Oct 30 22:44:31.486957 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.486918 5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv"] Oct 30 22:44:31.487388 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.487374 5508 cpu_manager.go:414] "containerMap: removing container" podUID="7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" containerName="maas-api" Oct 30 22:44:31.487440 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.487388 5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" containerName="maas-api" Oct 30 22:44:31.487475 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.487445 5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3c617c-e6b1-45e7-b8ab-3febc2d2dcaf" containerName="maas-api" Oct 30 22:44:31.490825 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.490806 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.547860 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.547829 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv"] Oct 30 22:44:31.555777 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.555746 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-sys\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.555936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.555789 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-proc\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.555936 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.555893 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-podres\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.556071 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.556051 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wh6k\" (UniqueName: \"kubernetes.io/projected/8d6072f9-7626-467d-80a9-19165699ffd8-kube-api-access-4wh6k\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.556136 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.556093 5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-lib-modules\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.594680 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.594657 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mff8v\"/\"kube-root-ca.crt\"" Oct 30 22:44:31.594791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.594705 5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mff8v\"/\"openshift-service-ca.crt\"" Oct 30 22:44:31.594791 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.594748 5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mff8v\"/\"default-dockercfg-nl99d\"" Oct 30 22:44:31.657508 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657482 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wh6k\" (UniqueName: \"kubernetes.io/projected/8d6072f9-7626-467d-80a9-19165699ffd8-kube-api-access-4wh6k\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657514 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-lib-modules\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657579 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-sys\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657635 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-sys\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657665 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-lib-modules\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657711 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-proc\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657749 5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-podres\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657786 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-podres\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.657851 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.657751 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8d6072f9-7626-467d-80a9-19165699ffd8-proc\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.718713 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.718682 5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wh6k\" (UniqueName: \"kubernetes.io/projected/8d6072f9-7626-467d-80a9-19165699ffd8-kube-api-access-4wh6k\") pod \"perf-node-gather-daemonset-nffzv\" (UID: \"8d6072f9-7626-467d-80a9-19165699ffd8\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:31.802938 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:31.802877 5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:32.175051 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:32.174901 5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv"] Oct 30 22:44:32.177837 ip-10-0-134-73 kubenswrapper[5508]: W1030 22:44:32.177812 5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d6072f9_7626_467d_80a9_19165699ffd8.slice/crio-061b60fa8bbd8d7cbd8f7dac6dd6485e6dbe82b0c3aa03ee3baf5139439668ce WatchSource:0}: Error finding container 061b60fa8bbd8d7cbd8f7dac6dd6485e6dbe82b0c3aa03ee3baf5139439668ce: Status 404 returned error can't find the container with id 061b60fa8bbd8d7cbd8f7dac6dd6485e6dbe82b0c3aa03ee3baf5139439668ce Oct 30 22:44:32.570470 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:32.570448 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-m84rz_f0f9640f-32ec-4f9f-8f4d-0648f1fd2f99/kuadrant-console-plugin/0.log" Oct 30 22:44:32.609226 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:32.609187 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" event={"ID":"8d6072f9-7626-467d-80a9-19165699ffd8","Type":"ContainerStarted","Data":"800653827b33189746a9b418959310a65073bf75e3d0aff38aabb02b75660aca"} Oct 30 22:44:32.609406 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:32.609250 5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" event={"ID":"8d6072f9-7626-467d-80a9-19165699ffd8","Type":"ContainerStarted","Data":"061b60fa8bbd8d7cbd8f7dac6dd6485e6dbe82b0c3aa03ee3baf5139439668ce"} Oct 30 22:44:32.609406 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:32.609347 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:32.723803 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:32.723753 5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" podStartSLOduration=1.7237379430000002 podStartE2EDuration="1.723737943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:32.722468677 +0000 UTC m=+3682.912006353" watchObservedRunningTime="2026-10-30 22:44:32.723737943 +0000 UTC m=+3682.913275611" Oct 30 22:44:38.624837 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:38.624807 5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-nffzv" Oct 30 22:44:40.373011 ip-10-0-134-73 kubenswrapper[5508]: E1030 22:44:40.372978 5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lbksg" containerName="extractor" Oct 30 22:44:41.261233 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:41.261196 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6499d46c6f-54lvw_26c4ad3d-7555-4bbf-834e-299ef37ffbcf/metrics-server/0.log" Oct 30 22:44:42.235779 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:42.235741 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:44:42.402327 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:42.402270 5508 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cggst_1a71cf03-8c4c-421f-ad1a-dd8735c1b2fc/konnectivity-agent/0.log" limit=1024 Oct 30 22:44:43.374393 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:43.374365 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qmc6j_3a0630d9-1d24-4533-ac24-c19bab683ae3/node-exporter/0.log" Oct 30 22:44:43.791884 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:43.791779 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qmc6j_3a0630d9-1d24-4533-ac24-c19bab683ae3/kube-rbac-proxy/0.log" Oct 30 22:44:44.208730 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:44.208704 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qmc6j_3a0630d9-1d24-4533-ac24-c19bab683ae3/init-textfile/0.log" Oct 30 22:44:45.884005 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:45.883968 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fbgs6_ccb804c3-d2fb-48bb-be14-3b1c6743f49f/kube-rbac-proxy-main/0.log" Oct 30 22:44:46.300101 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:46.300033 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fbgs6_ccb804c3-d2fb-48bb-be14-3b1c6743f49f/kube-rbac-proxy-self/0.log" Oct 30 22:44:46.751842 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:46.751801 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fbgs6_ccb804c3-d2fb-48bb-be14-3b1c6743f49f/openshift-state-metrics/0.log" Oct 30 22:44:47.197523 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:47.197497 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/prometheus/0.log" Oct 30 22:44:47.628272 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:47.628244 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/config-reloader/0.log" Oct 30 22:44:48.046797 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:48.046725 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/thanos-sidecar/0.log" Oct 30 22:44:48.468442 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:48.468416 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/kube-rbac-proxy-web/0.log" Oct 30 22:44:48.884783 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:48.884735 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/kube-rbac-proxy/0.log" Oct 30 22:44:49.302480 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:49.302410 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/kube-rbac-proxy-thanos/0.log" Oct 30 22:44:49.729574 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:49.729540 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_907c5318-2bc1-44cb-89ef-757db10b6009/init-config-reloader/0.log" Oct 30 22:44:50.151897 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:50.151866 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-gtjhm_50ebddb7-2e87-4a26-844e-bcb12013c7c2/prometheus-operator/0.log" Oct 30 22:44:50.566745 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:50.566665 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-gtjhm_50ebddb7-2e87-4a26-844e-bcb12013c7c2/kube-rbac-proxy/0.log" Oct 30 22:44:50.985667 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:50.985642 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cvlq5_6cb39913-45bd-4e03-8a53-e5a269238d4e/prometheus-operator-admission-webhook/0.log" Oct 30 22:44:51.411784 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:51.411750 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5659774bc7-pwf5k_e7e07f32-5013-4d0d-ac94-bd8759bfe082/telemeter-client/0.log" Oct 30 22:44:51.827831 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:51.827800 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5659774bc7-pwf5k_e7e07f32-5013-4d0d-ac94-bd8759bfe082/reload/0.log" Oct 30 22:44:52.252513 ip-10-0-134-73 kubenswrapper[5508]: I1030 22:44:52.252438 5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5659774bc7-pwf5k_e7e07f32-5013-4d0d-ac94-bd8759bfe082/kube-rbac-proxy/0.log"