Oct 30 19:15:52.769188 ip-10-0-140-127 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:15:53.163382 ip-10-0-140-127 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:53.163973 ip-10-0-140-127 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:15:53.163973 ip-10-0-140-127 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:53.163973 ip-10-0-140-127 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:15:53.169144 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169066 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169145 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169150 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169154 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169157 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169160 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169163 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169166 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169169 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169171 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169174 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169178 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169182 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169185 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.169182 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169189 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169192 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169195 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169198 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169201 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169203 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169206 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169208 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169210 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169218 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169221 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169223 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169226 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169229 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169232 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169236 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169240 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169243 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169245 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169248 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.169504 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169251 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169253 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169256 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169258 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169261 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169263 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169266 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169268 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169271 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169273 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169276 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169279 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169281 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169284 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169287 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169290 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169293 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169296 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169298 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.169967 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169300 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169303 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169305 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169308 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169311 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169313 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169315 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169318 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169320 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169323 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169326 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169328 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169331 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169333 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169336 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169338 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169341 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169343 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169346 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169348 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.170418 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169350 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169353 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169355 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169358 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169360 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169363 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169365 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169369 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169372 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169375 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169377 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169380 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169382 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169385 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169387 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169389 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169392 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169394 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169397 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.170925 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169399 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169402 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169404 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169406 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169409 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169414 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169416 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169419 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169421 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169423 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169426 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169428 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.169432 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170222 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170229 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170232 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170235 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170238 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170241 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.171363 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170244 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170247 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170250 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170253 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170256 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170258 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170261 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170264 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170266 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170269 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170271 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170274 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170276 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170279 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170281 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170284 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170287 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170289 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170292 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.171818 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170294 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170296 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170299 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170301 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170306 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170309 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170311 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170313 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170316 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170319 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170321 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170323 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170327 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170330 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170332 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170336 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170338 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170342 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170344 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.172302 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170347 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170349 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170352 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170354 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170357 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170359 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170361 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170371 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170374 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170377 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170379 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170382 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170384 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170386 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170389 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170393 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170397 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170400 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170402 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.172779 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170405 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170408 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170410 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170412 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170415 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170417 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170422 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170425 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170428 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170430 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170434 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170436 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170439 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170441 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170444 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170446 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170449 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170451 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170453 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170456 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.173271 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170458 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170461 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170463 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170465 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170468 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170472 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170474 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170477 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170480 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170482 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170484 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170487 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170489 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170492 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170494 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170497 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170499 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170502 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170506 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.173725 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170522 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170527 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.170531 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171052 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171058 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171061 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171065 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171067 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171070 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171073 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171075 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171078 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171080 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171083 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171086 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171088 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171091 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171093 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171097 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171100 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.174175 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171102 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171105 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171107 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171110 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171112 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171115 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171117 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171119 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171122 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171125 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171127 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171130 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171132 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171135 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171138 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171140 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171143 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171146 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171149 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.174665 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171152 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171156 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171159 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171162 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171166 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171169 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171172 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171174 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171177 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171179 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171182 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171187 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171190 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171192 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171195 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171197 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171199 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171202 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171204 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.175113 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171207 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171209 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171212 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171214 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171217 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171219 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171222 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171224 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171227 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171229 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171231 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171234 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171237 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171240 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171242 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171246 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171250 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171253 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171256 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171258 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.175556 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171261 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171264 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171267 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171270 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171274 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171277 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171279 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171281 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171284 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171286 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171289 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171291 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171294 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171296 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171298 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171301 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171304 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171306 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171309 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.176042 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171311 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171313 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171316 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171318 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171321 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171324 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171326 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171328 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171331 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171333 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171335 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.171338 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172216 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172227 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172233 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172247 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172252 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172256 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172260 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172264 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172268 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:15:53.176488 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172271 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172274 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172277 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172280 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172283 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172286 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172289 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172292 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172295 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172300 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172303 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172307 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172310 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172314 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172318 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172321 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172324 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172328 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172331 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172334 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172336 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172340 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172343 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172348 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:15:53.176992 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172350 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172353 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172356 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172359 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172362 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172367 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172370 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172373 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172376 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172379 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172383 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172386 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172389 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172392 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172395 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172398 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172401 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172404 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172406 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172409 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172412 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172415 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172418 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172421 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172424 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172427 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:15:53.177596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172430 2575 flags.go:64] FLAG: --help="false" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172433 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172437 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172440 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172442 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172446 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172449 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172452 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172454 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172457 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172460 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172463 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172466 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172468 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172471 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172474 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172477 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172479 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172482 2575 flags.go:64] FLAG: --lock-file="" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172485 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172487 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172490 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172495 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:15:53.178210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172498 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172501 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172504 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172507 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172510 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172513 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172516 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172520 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172523 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172527 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172538 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172541 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172544 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172546 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172551 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172554 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172557 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172569 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172572 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172574 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172577 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172581 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172583 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172586 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172589 2575 flags.go:64] FLAG: --port="10250" Oct 30 19:15:53.178843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172592 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172595 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-06955bed59c5b342a" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172598 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172600 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172603 2575 flags.go:64] FLAG: --register-node="true" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172606 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172610 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172612 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172615 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172617 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172621 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172624 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172627 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172630 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172632 2575 flags.go:64] FLAG: --runonce="false" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172635 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172638 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172641 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172644 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172647 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172649 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172654 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172657 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172660 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172663 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:15:53.179917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172666 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172669 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172672 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172675 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172677 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172682 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172685 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172688 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172692 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172695 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172697 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172700 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172703 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172705 2575 flags.go:64] FLAG: --v="2" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172710 2575 flags.go:64] FLAG: --version="false" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172714 2575 flags.go:64] FLAG: --vmodule="" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172718 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.172721 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172855 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172859 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172861 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172864 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172866 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.181069 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172869 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172872 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172874 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172890 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172893 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172897 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172899 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172902 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172904 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172907 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172911 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172913 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172916 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172918 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172921 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172923 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172926 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172929 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172932 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172934 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.182055 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172937 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172940 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172942 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172945 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172947 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172949 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172952 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172954 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172957 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172960 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172962 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172965 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172968 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172970 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172972 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172975 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172978 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172982 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172985 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172987 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.182978 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172989 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172992 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172996 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.172999 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173001 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173003 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173006 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173008 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173011 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173013 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173016 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173018 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173021 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173023 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173025 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173028 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173030 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173033 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173035 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.183572 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173037 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173040 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173042 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173045 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173047 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173050 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173052 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173054 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173057 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173060 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173065 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173069 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173072 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173075 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173078 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173081 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173083 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173086 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173088 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.184386 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173091 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173094 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173096 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173099 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173101 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173104 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173106 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173109 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173111 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173113 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173116 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173118 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173122 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173125 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173129 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173132 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173134 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173137 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173139 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.185231 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173141 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173144 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173146 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173148 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.173176 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173212 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173216 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173219 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173221 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173224 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173227 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173230 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173232 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173235 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173238 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173240 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.185693 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173242 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173245 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173248 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173250 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173252 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173255 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173257 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173260 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173262 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173264 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173267 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173270 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173272 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173275 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173277 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173280 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173282 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173285 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173288 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173290 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.186160 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173294 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173297 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173300 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173303 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173305 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173307 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173311 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173314 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173316 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173319 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173322 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173324 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173326 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173329 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173331 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173334 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173336 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173339 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173341 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.186621 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173344 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173346 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173349 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173351 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173353 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173356 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173358 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173361 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173363 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173365 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173368 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173370 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173373 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173377 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173379 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173382 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173384 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173387 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173389 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.187238 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173393 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173395 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173398 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173400 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173403 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173405 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173408 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173411 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173413 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173417 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173421 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173423 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173426 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173428 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173430 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173433 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173435 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173438 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173440 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.187790 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173443 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173445 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173448 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173450 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173452 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173455 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173457 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173461 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173463 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173465 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173468 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173474 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173477 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173481 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173484 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173486 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173489 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.173491 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.181148 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:15:53.188316 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.181168 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.181191 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.181198 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181281 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181289 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181296 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181301 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181306 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181310 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181315 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181319 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181323 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181327 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181331 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181335 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181340 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181344 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181349 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181353 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.188755 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181357 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181361 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181365 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181369 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181374 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181378 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181382 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181386 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181390 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181394 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181399 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181403 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181407 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181413 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181418 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181422 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181426 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181430 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181435 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181439 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.189281 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181443 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181447 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181451 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181455 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181459 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181464 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181468 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181472 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181477 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181481 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181485 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181489 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181495 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181503 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181509 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181513 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181518 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181523 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181527 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.189829 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181531 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181535 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181539 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181544 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181548 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181553 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181557 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181563 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181568 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181573 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181577 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181581 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181585 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181589 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181593 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181598 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181602 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181606 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181610 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181614 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.190293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181618 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181622 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181626 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181631 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181635 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181639 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181644 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181648 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181652 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181656 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181660 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181664 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181668 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181673 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181677 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181681 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181685 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181690 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181694 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.191115 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181698 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181703 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181709 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181714 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181720 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181726 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181731 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181735 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181739 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181743 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181747 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.181751 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.181790 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182034 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182043 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.191611 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182047 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182052 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182056 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182059 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182063 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182067 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182071 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182075 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182079 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182082 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182086 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182091 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182095 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182099 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182103 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182107 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182112 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182116 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182120 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.192228 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182125 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182129 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182133 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182137 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182141 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182145 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182149 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182153 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182157 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182161 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182165 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182169 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182173 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182177 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182181 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182184 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182189 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182193 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182197 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.192680 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182201 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182207 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182214 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182218 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182222 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182226 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182231 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182235 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182239 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182243 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182247 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182251 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182255 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182260 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182264 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182269 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182273 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182277 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182281 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.193164 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182285 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182289 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182293 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182297 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182301 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182305 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182309 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182313 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182317 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182321 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182325 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182329 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182333 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182337 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182341 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182345 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182351 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182355 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182359 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182363 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.193606 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182367 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182371 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182375 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182379 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182383 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182387 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182391 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182396 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182401 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182405 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182409 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182413 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182420 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182425 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182430 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182434 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182438 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182442 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182446 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.194137 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182450 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182454 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182458 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182462 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182466 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182470 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182474 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182478 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.182516 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182568 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182574 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182578 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182582 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182586 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182590 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182594 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:15:53.194566 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182599 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182605 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182612 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182616 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182620 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182624 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182629 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182633 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182637 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182641 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182645 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182650 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182654 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182658 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182663 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182667 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182671 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182675 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182679 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:15:53.194948 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182683 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182692 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182696 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182701 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182705 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182709 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182713 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182717 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182721 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182726 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182730 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182734 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182738 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182743 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182747 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182751 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182755 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182760 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182764 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182768 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:15:53.195390 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182773 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182777 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182782 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182786 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182790 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182794 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182798 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182802 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182807 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182812 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182816 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182820 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182826 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182833 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182838 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182843 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182847 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182852 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182856 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:15:53.195844 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182860 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182864 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182868 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182872 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182876 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182898 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182903 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182907 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182911 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182917 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182921 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182925 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182929 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182933 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182937 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182942 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182946 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182950 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182954 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182958 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:15:53.196351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182962 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182966 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182970 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182974 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182978 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182982 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182988 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182991 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182995 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.182999 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183003 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183007 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183011 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183015 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183019 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183023 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183027 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183031 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183035 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:15:53.196953 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183039 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:53.183043 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.183118 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.187262 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.188206 2575 server.go:1014] "Starting client certificate rotation" Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.188304 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.188348 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:15:53.197684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.191539 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:53.199160 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.199142 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:15:53.210114 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.210095 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:15:53.212867 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.212853 2575 log.go:25] "Validated CRI v1 image API" Oct 30 19:15:53.213669 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.213643 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:15:53.219047 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.219022 2575 fs.go:135] Filesystem UUIDs: map[1cad5309-31dc-4079-88f9-ceb757267700:/dev/nvme0n1p3 37ba7ff5-7b29-4005-bf75-e14279c53fb4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 19:15:53.219096 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.219041 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:15:53.219909 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.219892 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:15:53.224041 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.223939 2575 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:53.222628933 +0000 UTC m=+0.335830515 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099949 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe4d5f95420627167c9c68aaf947d SystemUUID:ec2fe4d5-f954-2062-7167-c9c68aaf947d BootID:ba493a55-7836-4585-8fb6-e1c037fbdb22 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cc:b8:a0:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cc:b8:a0:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:6f:4a:b7:04:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:15:53.224041 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.224040 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:15:53.224151 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.224135 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:15:53.226794 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.226775 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:15:53.226941 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.226796 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-127.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:15:53.226988 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.226943 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:15:53.226988 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.226951 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:15:53.226988 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.226982 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:53.227062 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.226998 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:15:53.227670 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.227661 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:15:53.231036 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231025 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:15:53.231157 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231148 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:15:53.231191 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231178 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:15:53.231191 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231187 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:15:53.231242 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231197 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:15:53.231242 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231208 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:15:53.231242 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231216 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:15:53.231497 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.231486 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:53.233691 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.233672 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:15:53.238003 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.237991 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:15:53.238836 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.238824 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:15:53.238890 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.238855 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:15:53.239020 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239009 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:15:53.239048 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239029 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:15:53.239048 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239041 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:15:53.239113 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239048 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:15:53.239113 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239058 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:15:53.239113 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239068 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:15:53.239502 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239493 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:15:53.239529 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239504 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:15:53.239529 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239518 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:15:53.239529 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239524 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:15:53.239694 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239540 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:15:53.239694 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.239548 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:15:53.240192 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.240181 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:15:53.240192 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.240192 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:15:53.242265 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.242243 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-127.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:15:53.242519 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.242500 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfb6r" Oct 30 19:15:53.244104 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.244091 2575 server.go:1263] "Started kubelet" Oct 30 19:15:53.244201 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.244155 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:15:53.244260 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.244185 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:15:53.244294 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.244282 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:15:53.249150 ip-10-0-140-127 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:15:53.250241 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.249537 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:15:53.250812 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.250783 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:15:53.253931 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.253913 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:15:53.254333 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.254307 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfb6r" Oct 30 19:15:53.255179 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.255156 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:15:53.255273 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.255211 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-127.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:15:53.256268 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.255377 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-127.us-east-2.compute.internal.18e364a8e8a603d8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-127.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-127.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:15:53.244066776 +0000 UTC m=+0.357268359,LastTimestamp:2026-10-30 19:15:53.244066776 +0000 UTC m=+0.357268359,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-127.us-east-2.compute.internal,}" Oct 30 19:15:53.257689 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.257674 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:15:53.258048 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.258026 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:15:53.258180 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.258160 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.258804 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.258787 2575 factory.go:55] Registering systemd factory Oct 30 19:15:53.258892 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.258809 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:15:53.258960 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.258946 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:15:53.258960 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.258947 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:15:53.259037 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.258984 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:15:53.259098 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259086 2575 factory.go:153] Registering CRI-O factory Oct 30 19:15:53.259131 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259105 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 19:15:53.259171 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259159 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:15:53.259234 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259093 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:15:53.259234 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259183 2575 factory.go:103] Registering Raw factory Oct 30 19:15:53.259234 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259191 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:15:53.259234 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.259199 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 19:15:53.260083 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.260064 2575 manager.go:319] Starting recovery of all containers Oct 30 19:15:53.263643 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.263623 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:53.266192 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.266168 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-127.us-east-2.compute.internal\" not found" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.270469 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.270450 2575 manager.go:324] Recovery completed Oct 30 19:15:53.274421 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.274409 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.276028 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276012 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.276129 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276038 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.276129 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276052 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.276620 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276608 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:15:53.276620 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276619 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:15:53.276691 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276632 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.276691 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.276636 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:15:53.278226 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.278204 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.278336 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.278243 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.278336 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.278259 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.279172 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.279155 2575 policy_none.go:50] "Start" Oct 30 19:15:53.279172 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.279173 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:15:53.279273 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.279190 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:15:53.281088 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.281075 2575 policy_none.go:44] "Start" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.329448 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.331803 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.331851 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.331860 2575 server.go:87] "Starting device plugin registration server" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.332115 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.332126 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.332225 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.332306 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.332316 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.332851 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.332909 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.340330 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.341615 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.341633 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.341654 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.341697 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:15:53.345626 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.344417 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:53.433003 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.432934 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.433985 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.433964 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.434078 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.433996 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.434078 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.434008 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.434078 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.434031 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.440508 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.440493 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.440551 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.440515 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-127.us-east-2.compute.internal\": node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.442763 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.442748 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal"] Oct 30 19:15:53.442804 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.442798 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.443776 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.443760 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.443843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.443789 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.443843 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.443804 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.446072 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446059 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.446229 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446215 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.446270 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446246 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.446796 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446775 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.446876 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446797 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.446876 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446819 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.446876 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446828 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.446876 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446802 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.447095 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.446909 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.449093 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.449077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.449173 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.449108 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:15:53.449853 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.449837 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:15:53.449929 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.449868 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:15:53.449929 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.449901 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:15:53.456875 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.456860 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.472400 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.472385 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.474414 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.474400 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.557403 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.557375 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.560664 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.560649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.560730 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.560673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4860c9cfd0c091391ec52fd8f7245c90-config\") pod \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"4860c9cfd0c091391ec52fd8f7245c90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.560730 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.560695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.658362 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.658331 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.661598 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.661583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.661651 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.661606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.661651 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.661624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4860c9cfd0c091391ec52fd8f7245c90-config\") pod \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"4860c9cfd0c091391ec52fd8f7245c90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.661722 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.661681 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4860c9cfd0c091391ec52fd8f7245c90-config\") pod \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"4860c9cfd0c091391ec52fd8f7245c90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.661722 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.661687 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.661722 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.661687 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.759125 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.759049 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.775353 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.775335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.777490 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:53.777474 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:53.860242 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.860085 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:53.960611 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:53.960572 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.061125 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.061054 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.161615 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.161589 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.187913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.187877 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:15:54.188318 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.188005 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:15:54.188318 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.188059 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:15:54.260598 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.260547 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:54 +0000 UTC" deadline="2028-06-23 11:59:01.017544613 +0000 UTC" Oct 30 19:15:54.260598 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.260597 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14440h43m6.756950367s" Oct 30 19:15:54.262544 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.262522 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.363483 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.363455 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.430721 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:54.430678 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4860c9cfd0c091391ec52fd8f7245c90.slice/crio-856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c WatchSource:0}: Error finding container 856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c: Status 404 returned error can't find the container with id 856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c Oct 30 19:15:54.435608 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.435584 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:15:54.435706 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:54.435686 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62885df979c25d07a503fc33848fc4ff.slice/crio-d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94 WatchSource:0}: Error finding container d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94: Status 404 returned error can't find the container with id d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94 Oct 30 19:15:54.464281 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.464262 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.564914 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.564813 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.665315 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:54.665283 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:15:54.694206 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.694188 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:54.734330 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.734307 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:54.758394 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.758372 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:54.769561 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.769543 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:54.770633 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.770621 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:15:54.782080 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:54.782066 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:15:55.024487 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.024409 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:55.189507 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.189461 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:15:55.233019 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.232716 2575 apiserver.go:52] "Watching apiserver" Oct 30 19:15:55.248664 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.248636 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:15:55.249010 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.248987 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-52x9m","openshift-multus/multus-additional-cni-plugins-gh6sj","openshift-ovn-kubernetes/ovnkube-node-nbd9r","openshift-dns/node-resolver-kxt77","openshift-image-registry/node-ca-4wfl7","openshift-network-diagnostics/network-check-target-9hzhx","openshift-network-operator/iptables-alerter-vf6wx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th","openshift-cluster-node-tuning-operator/tuned-8r4wz","openshift-multus/network-metrics-daemon-468t5","kube-system/global-pull-secret-syncer-f6wtd","kube-system/konnectivity-agent-hcv7r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal"] Oct 30 19:15:55.251915 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.251637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.253830 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.253809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.255596 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.255573 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:15:55.255690 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.255576 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:15:55.255690 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.255603 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:15:55.255690 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.255601 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:15:55.255786 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.255766 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wctlt\"" Oct 30 19:15:55.256288 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.256269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.256471 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.256449 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:15:55.256739 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.256722 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:15:55.257001 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.256983 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzjv5\"" Oct 30 19:15:55.258237 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.258219 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:15:55.258553 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.258531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:55.259201 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259159 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:15:55.259201 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259189 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:15:55.259861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259573 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:15:55.259861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259651 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:15:55.259861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259746 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qmpzj\"" Oct 30 19:15:55.259861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259799 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:15:55.259861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.259816 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:15:55.260829 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.260809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:55.261724 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.261685 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:15:55.261819 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.261767 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:15:55.261905 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.261836 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6xn8\"" Oct 30 19:15:55.263001 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.262985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:15:55.263342 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:55.263316 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:15:55.266428 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.266406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:55.270264 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:55.270607 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-netns\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-node-log\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-netd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-config\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-system-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.270684 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cnibin\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.270874 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-kubelet\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270874 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-etc-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270874 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270874 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-script-lib\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.270874 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-conf-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270899 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-systemd-units\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-ovn\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.270988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovn-node-metrics-cert\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zflc8\" (UniqueName: \"kubernetes.io/projected/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-kube-api-access-zflc8\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-host\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271049 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-k8s-cni-cncf-io\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-netns\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-multus\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.271136 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfb6\" (UniqueName: \"kubernetes.io/projected/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-kube-api-access-vkfb6\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cnibin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-daemon-config\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-multus-certs\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-log-socket\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-bin\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32f31184-b3c4-4a0e-8976-1d91195570e8-hosts-file\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2ww2\" (UniqueName: \"kubernetes.io/projected/32f31184-b3c4-4a0e-8976-1d91195570e8-kube-api-access-t2ww2\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-serviceca\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cni-binary-copy\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-bin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-var-lib-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-env-overrides\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.271674 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6wl9\" (UniqueName: \"kubernetes.io/projected/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-kube-api-access-m6wl9\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f31184-b3c4-4a0e-8976-1d91195570e8-tmp-dir\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-kubelet\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-etc-kubernetes\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnsw\" (UniqueName: \"kubernetes.io/projected/0e6ea1e8-21c6-48fb-beb1-7246750874a4-kube-api-access-7hnsw\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-slash\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-systemd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-system-cni-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-socket-dir-parent\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.272398 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.271712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-hostroot\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:55.272825 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.272597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:55.276045 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.275999 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:55.276133 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:55.276069 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:15:55.278389 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.278368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:55.278475 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:55.278430 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:15:55.280975 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:55.280953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.524617 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.524593 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7ctgd" Oct 30 19:15:56.540471 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.539480 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7ctgd" Oct 30 19:15:56.578159 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.578108 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"62885df979c25d07a503fc33848fc4ff","Type":"ContainerStarted","Data":"d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94"} Oct 30 19:15:56.579304 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.579277 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"4860c9cfd0c091391ec52fd8f7245c90","Type":"ContainerStarted","Data":"856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c"} Oct 30 19:15:56.588002 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.587976 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:15:56.588206 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588069 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:56.588206 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588138 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:15:56.588378 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588217 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:15:56.588378 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588312 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:15:56.588505 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588437 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7s42\"" Oct 30 19:15:56.588610 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588549 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:15:56.588660 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588612 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:15:56.588709 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588701 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrdg\"" Oct 30 19:15:56.588793 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588778 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:15:56.588887 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588873 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9tgzd\"" Oct 30 19:15:56.588985 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.588957 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:15:56.589098 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.589044 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:15:56.589163 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.589130 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mglp5\"" Oct 30 19:15:56.589214 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.589194 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:15:56.589279 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.589264 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:15:56.589394 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.589370 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xph5z\"" Oct 30 19:15:56.589535 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.589520 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:15:56.591835 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.591802 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:15:56.604019 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.603950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-etc-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.604019 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.603995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-tmp\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.604210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-etc-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.604210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jkz\" (UniqueName: \"kubernetes.io/projected/3b0c721a-72d7-44ce-af65-457dc171bfca-kube-api-access-27jkz\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:56.604210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-conf-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.604210 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.604377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-ovn\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.604377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-dbus\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.604377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604272 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-conf-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.604377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-ovn\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.604377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-device-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysconfig\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-sys\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-netns\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-multus\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604573 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-multus\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.604653 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604631 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-netns\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-run\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-daemon-config\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-multus-certs\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604869 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-log-socket\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-konnectivity-ca\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-iptables-alerter-script\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.604985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-host\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32f31184-b3c4-4a0e-8976-1d91195570e8-hosts-file\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-socket-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.605044 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-serviceca\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-var-lib-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fdwt\" (UniqueName: \"kubernetes.io/projected/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-kube-api-access-8fdwt\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-etc-selinux\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605122 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-conf\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f31184-b3c4-4a0e-8976-1d91195570e8-tmp-dir\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-kubelet\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-multus-certs\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-etc-kubernetes\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-slash\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-var-lib-kubelet\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-system-cni-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-socket-dir-parent\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-hostroot\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.605913 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-netns\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-netd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-kubelet-config\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605375 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-daemon-config\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-lib-modules\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-system-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cnibin\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605436 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-slash\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-script-lib\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-agent-certs\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-systemd-units\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovn-node-metrics-cert\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-host-slash\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-sys-fs\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflc8\" (UniqueName: \"kubernetes.io/projected/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-kube-api-access-zflc8\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.606704 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-host\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-k8s-cni-cncf-io\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-modprobe-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605639 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-serviceca\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-etc-tuned\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfb6\" (UniqueName: \"kubernetes.io/projected/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-kube-api-access-vkfb6\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605703 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-log-socket\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cnibin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605806 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32f31184-b3c4-4a0e-8976-1d91195570e8-hosts-file\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605824 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-socket-dir-parent\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605864 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-host\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605861 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-netns\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605888 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-systemd-units\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605903 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cnibin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.605934 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-netd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.607339 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606002 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-system-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606028 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cnibin\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606049 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606146 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606201 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-kubelet\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-bin\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b22n\" (UniqueName: \"kubernetes.io/projected/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kube-api-access-2b22n\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-kubernetes\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-systemd\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606280 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-hostroot\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8s7\" (UniqueName: \"kubernetes.io/projected/b70be47d-c307-4f47-b075-89a53633411d-kube-api-access-qs8s7\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606347 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-etc-kubernetes\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606377 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-system-cni-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606430 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-k8s-cni-cncf-io\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606465 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-bin\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ww2\" (UniqueName: \"kubernetes.io/projected/32f31184-b3c4-4a0e-8976-1d91195570e8-kube-api-access-t2ww2\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606510 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-var-lib-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cni-binary-copy\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608178 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-bin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-bin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-env-overrides\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-script-lib\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6wl9\" (UniqueName: \"kubernetes.io/projected/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-kube-api-access-m6wl9\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606740 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f31184-b3c4-4a0e-8976-1d91195570e8-tmp-dir\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607005 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cni-binary-copy\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607054 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-env-overrides\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.606760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607241 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnsw\" (UniqueName: \"kubernetes.io/projected/0e6ea1e8-21c6-48fb-beb1-7246750874a4-kube-api-access-7hnsw\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-systemd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-node-log\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-config\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607360 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-systemd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-registration-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-kubelet\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.608949 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607484 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-node-log\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.609804 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-kubelet\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.609804 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.607552 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.609804 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.609016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-config\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.611923 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.611902 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovn-node-metrics-cert\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.617947 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.617927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflc8\" (UniqueName: \"kubernetes.io/projected/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-kube-api-access-zflc8\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.618470 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.618452 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfb6\" (UniqueName: \"kubernetes.io/projected/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-kube-api-access-vkfb6\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.618921 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.618887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ww2\" (UniqueName: \"kubernetes.io/projected/32f31184-b3c4-4a0e-8976-1d91195570e8-kube-api-access-t2ww2\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.620030 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.620010 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6wl9\" (UniqueName: \"kubernetes.io/projected/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-kube-api-access-m6wl9\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.620825 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.620804 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnsw\" (UniqueName: \"kubernetes.io/projected/0e6ea1e8-21c6-48fb-beb1-7246750874a4-kube-api-access-7hnsw\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.708708 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.708708 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-modprobe-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-etc-tuned\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b22n\" (UniqueName: \"kubernetes.io/projected/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kube-api-access-2b22n\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-kubernetes\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-systemd\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708830 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708878 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-modprobe-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708890 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.708925 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708903 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-kubernetes\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.708945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-systemd\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8s7\" (UniqueName: \"kubernetes.io/projected/b70be47d-c307-4f47-b075-89a53633411d-kube-api-access-qs8s7\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-registration-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-tmp\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jkz\" (UniqueName: \"kubernetes.io/projected/3b0c721a-72d7-44ce-af65-457dc171bfca-kube-api-access-27jkz\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-dbus\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-device-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysconfig\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-sys\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709255 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-run\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-konnectivity-ca\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-iptables-alerter-script\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-host\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709313 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-device-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-socket-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fdwt\" (UniqueName: \"kubernetes.io/projected/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-kube-api-access-8fdwt\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-etc-selinux\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-conf\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-var-lib-kubelet\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-kubelet-config\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-dbus\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-lib-modules\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-agent-certs\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.709700 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-host-slash\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-sys-fs\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709637 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-socket-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-sys-fs\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709678 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysconfig\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709709 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-sys\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.709736 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.709794 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.209778061 +0000 UTC m=+3.091446915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.709842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-registration-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710165 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-etc-selinux\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.710284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-conf\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710294 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-var-lib-kubelet\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-kubelet-config\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.710392 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-konnectivity-ca\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-iptables-alerter-script\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.710441 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.210430142 +0000 UTC m=+3.092098984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710478 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-host\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710516 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-run\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-host-slash\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.710672 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.710578 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-lib-modules\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.711974 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.711945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-tmp\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.712968 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.712930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-etc-tuned\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.714060 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.714019 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-agent-certs\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.718349 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.718198 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:56.718349 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.718227 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:56.718349 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.718242 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:56.718349 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:56.718321 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.218301511 +0000 UTC m=+3.099970383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:56.721139 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.721110 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fdwt\" (UniqueName: \"kubernetes.io/projected/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-kube-api-access-8fdwt\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.723144 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.723118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jkz\" (UniqueName: \"kubernetes.io/projected/3b0c721a-72d7-44ce-af65-457dc171bfca-kube-api-access-27jkz\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:56.726170 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.725771 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b22n\" (UniqueName: \"kubernetes.io/projected/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kube-api-access-2b22n\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.726170 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.726121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8s7\" (UniqueName: \"kubernetes.io/projected/b70be47d-c307-4f47-b075-89a53633411d-kube-api-access-qs8s7\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.799179 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.799092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-52x9m" Oct 30 19:15:56.811764 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.811731 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9e4ecb_e4fb_4d66_99a4_b17cf8f4c0cc.slice/crio-ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576 WatchSource:0}: Error finding container ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576: Status 404 returned error can't find the container with id ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576 Oct 30 19:15:56.825455 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.825429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:15:56.828188 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.828164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:15:56.836068 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.836032 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a35f2ba_0f4b_41e8_b3b5_6ba6ff4d62a7.slice/crio-c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860 WatchSource:0}: Error finding container c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860: Status 404 returned error can't find the container with id c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860 Oct 30 19:15:56.839286 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.839202 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6ea1e8_21c6_48fb_beb1_7246750874a4.slice/crio-c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf WatchSource:0}: Error finding container c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf: Status 404 returned error can't find the container with id c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf Oct 30 19:15:56.844054 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.844031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kxt77" Oct 30 19:15:56.847840 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.847819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:15:56.850111 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.850077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:15:56.857213 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.857179 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f31184_b3c4_4a0e_8976_1d91195570e8.slice/crio-6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725 WatchSource:0}: Error finding container 6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725: Status 404 returned error can't find the container with id 6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725 Oct 30 19:15:56.858059 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.858040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:15:56.859240 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.859192 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ce0b21_0196_4d7c_9b36_99ec47e7d057.slice/crio-09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490 WatchSource:0}: Error finding container 09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490: Status 404 returned error can't find the container with id 09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490 Oct 30 19:15:56.862387 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.862354 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:15:56.863673 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.863650 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00a389a_2d47_4c51_9e90_85e9ffe3c9d6.slice/crio-2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18 WatchSource:0}: Error finding container 2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18: Status 404 returned error can't find the container with id 2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18 Oct 30 19:15:56.865718 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:56.865699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:15:56.878351 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.878322 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70be47d_c307_4f47_b075_89a53633411d.slice/crio-a351306416be9f883844389db318983b12964a5ff58955a7c3867070e8142bb6 WatchSource:0}: Error finding container a351306416be9f883844389db318983b12964a5ff58955a7c3867070e8142bb6: Status 404 returned error can't find the container with id a351306416be9f883844389db318983b12964a5ff58955a7c3867070e8142bb6 Oct 30 19:15:56.879738 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:15:56.879714 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950c0201_b4d4_4b5a_9eb2_c0deb0efe1d4.slice/crio-1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643 WatchSource:0}: Error finding container 1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643: Status 404 returned error can't find the container with id 1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643 Oct 30 19:15:57.212525 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.212505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:57.212599 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.212539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:57.212649 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.212633 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:57.212694 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.212685 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:15:58.212672964 +0000 UTC m=+4.094341805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:57.212738 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.212633 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:57.212784 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.212772 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:58.212758707 +0000 UTC m=+4.094427545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:57.313464 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.313431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:15:57.313588 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.313558 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:57.313588 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.313583 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:57.313678 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.313596 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:57.313678 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.313659 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:58.313641226 +0000 UTC m=+4.195310067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:57.542989 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.542477 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:56 +0000 UTC" deadline="2028-05-18 13:50:22.001563123 +0000 UTC" Oct 30 19:15:57.542989 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.542888 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13578h34m24.458681433s" Oct 30 19:15:57.574061 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.574032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:57.574207 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:57.574169 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:15:57.592023 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.591987 2575 generic.go:356] "Generic (PLEG): container finished" podID="62885df979c25d07a503fc33848fc4ff" containerID="fdbcf6a69de0734f0724543db584efde6bfcea114b42e7b7af3a8890597d9d29" exitCode=0 Oct 30 19:15:57.592166 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.592078 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"62885df979c25d07a503fc33848fc4ff","Type":"ContainerDied","Data":"fdbcf6a69de0734f0724543db584efde6bfcea114b42e7b7af3a8890597d9d29"} Oct 30 19:15:57.604729 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.604658 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hcv7r" event={"ID":"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4","Type":"ContainerStarted","Data":"1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643"} Oct 30 19:15:57.609679 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.609619 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4wfl7" event={"ID":"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6","Type":"ContainerStarted","Data":"2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18"} Oct 30 19:15:57.623086 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.623009 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kxt77" event={"ID":"32f31184-b3c4-4a0e-8976-1d91195570e8","Type":"ContainerStarted","Data":"6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725"} Oct 30 19:15:57.639581 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.639557 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52x9m" event={"ID":"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc","Type":"ContainerStarted","Data":"ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576"} Oct 30 19:15:57.649669 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.649613 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"4860c9cfd0c091391ec52fd8f7245c90","Type":"ContainerStarted","Data":"ca8785ade9db83bdcb904d9b2040b82211c3fb67e2f3b86802e011301dff3139"} Oct 30 19:15:57.653144 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.652445 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" event={"ID":"b70be47d-c307-4f47-b075-89a53633411d","Type":"ContainerStarted","Data":"a351306416be9f883844389db318983b12964a5ff58955a7c3867070e8142bb6"} Oct 30 19:15:57.665527 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.664035 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" podStartSLOduration=1.664019428 podStartE2EDuration="1.664019428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:57.663720191 +0000 UTC m=+3.545389040" watchObservedRunningTime="2026-10-30 19:15:57.664019428 +0000 UTC m=+3.545688276" Oct 30 19:15:57.678643 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.678506 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" event={"ID":"e00ba2b0-311f-4ab1-9e26-e5020d81c211","Type":"ContainerStarted","Data":"d73776132b56f4a6ae32955eb64f598aee379e4ab6599c05a133466ee34b3ed0"} Oct 30 19:15:57.686585 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.686551 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490"} Oct 30 19:15:57.694169 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.694106 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerStarted","Data":"c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf"} Oct 30 19:15:57.698504 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:57.698440 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vf6wx" event={"ID":"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7","Type":"ContainerStarted","Data":"c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860"} Oct 30 19:15:58.231970 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.231735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:58.231970 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.231801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:58.231970 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.231953 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:58.232260 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.232003 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:00.231988877 +0000 UTC m=+6.113657715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:15:58.232260 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.232049 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:58.232260 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.232073 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:16:00.232065855 +0000 UTC m=+6.113734693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:15:58.332792 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.332754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:15:58.332998 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.332981 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:15:58.333076 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.333007 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:15:58.333076 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.333022 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:58.333166 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.333081 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:00.333063221 +0000 UTC m=+6.214732054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:15:58.543152 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.543063 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:56 +0000 UTC" deadline="2028-04-30 23:40:28.710317037 +0000 UTC" Oct 30 19:15:58.543152 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.543108 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13156h24m30.167213968s" Oct 30 19:15:58.577447 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.574546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:15:58.577447 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.574697 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:15:58.577447 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.575209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:15:58.577447 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:58.575320 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:15:58.719851 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:58.719813 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"62885df979c25d07a503fc33848fc4ff","Type":"ContainerStarted","Data":"c02648b332dd266084e43fe91bb4f63f0e6dfaab17a2faf1e5c1ea9ff09286a8"} Oct 30 19:15:59.574330 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:59.574304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:15:59.574759 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:15:59.574450 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:15:59.736807 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:59.736770 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="ec84063e2c92ac255829233e073c913cdfbe96bd2783e40772be160730fd2a42" exitCode=0 Oct 30 19:15:59.737819 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:59.737791 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerDied","Data":"ec84063e2c92ac255829233e073c913cdfbe96bd2783e40772be160730fd2a42"} Oct 30 19:15:59.760511 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:15:59.760458 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" podStartSLOduration=4.760440447 podStartE2EDuration="4.760440447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:58.735466656 +0000 UTC m=+4.617135503" watchObservedRunningTime="2026-10-30 19:15:59.760440447 +0000 UTC m=+5.642109302" Oct 30 19:16:00.258250 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:00.257193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:00.258250 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:00.257373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:00.258250 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.257534 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:00.258250 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.257598 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:04.257579321 +0000 UTC m=+10.139248166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:00.258250 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.258010 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:00.258250 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.258060 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:16:04.258044674 +0000 UTC m=+10.139713523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:00.358396 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:00.358355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:00.358585 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.358564 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:00.358672 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.358597 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:00.358672 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.358611 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:00.358672 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.358669 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:04.358651277 +0000 UTC m=+10.240320122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:00.581436 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:00.580535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:00.581436 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.580707 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:00.581436 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:00.581172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:00.581436 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:00.581305 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:01.573814 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:01.573783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:01.573985 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:01.573942 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:01.751803 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:01.751753 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hcv7r" event={"ID":"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4","Type":"ContainerStarted","Data":"7c913fd386ce70e128b700ce026697cb0502ce850859b4b0314a3fe93f62ad9d"} Oct 30 19:16:01.950048 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:01.949877 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:16:01.950828 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:01.950615 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:16:01.966314 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:01.966266 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hcv7r" podStartSLOduration=3.80859128 podStartE2EDuration="7.966249168s" totalImagesPullingTime="4.157657888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:01.769123477 +0000 UTC m=+7.650792334" watchObservedRunningTime="2026-10-30 19:16:01.966249168 +0000 UTC m=+7.847918016" Oct 30 19:16:02.575745 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:02.575723 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:02.575876 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:02.575855 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:02.576215 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:02.576197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:02.576309 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:02.576286 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:02.759589 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:02.759554 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" event={"ID":"e00ba2b0-311f-4ab1-9e26-e5020d81c211","Type":"ContainerStarted","Data":"ca6388204263bfa87df222d0de370f61540a8f19f55d7a9573a8d0179383f56b"} Oct 30 19:16:02.763098 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:02.763071 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4wfl7" event={"ID":"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6","Type":"ContainerStarted","Data":"c45b4824aab19bf70340eeef14e669969e1d186871aa86edc8cdb9627876cadf"} Oct 30 19:16:02.778328 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:02.778290 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4wfl7" podStartSLOduration=3.564749756 podStartE2EDuration="8.778279123s" totalImagesPullingTime="5.213529367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:02.777993604 +0000 UTC m=+8.659662453" watchObservedRunningTime="2026-10-30 19:16:02.778279123 +0000 UTC m=+8.659947970" Oct 30 19:16:03.574909 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:03.574376 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:03.574909 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:03.574537 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:03.766861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:03.766488 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:16:04.290723 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.290694 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:16:04.292893 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.292675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:04.292893 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.292723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:04.292893 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.292807 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:04.292893 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.292838 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:04.292893 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.292872 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.292852469 +0000 UTC m=+18.174521297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:04.292893 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.292892 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.29288231 +0000 UTC m=+18.174551136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:04.393885 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.393699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:04.394040 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.393908 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:04.394040 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.393933 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:04.394040 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.393947 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:04.394040 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.394004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:12.393986529 +0000 UTC m=+18.275655361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:04.571223 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.570495 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:16:04.290719672Z","UUID":"09eeeca5-0175-4290-9abc-e8c8939799c2","Handler":null,"Name":"","Endpoint":""} Oct 30 19:16:04.576965 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.576924 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:16:04.576965 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.576968 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:16:04.577394 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.577377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:04.577526 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.577501 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:04.577679 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.577662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:04.577763 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:04.577745 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:04.774509 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.774399 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" event={"ID":"e00ba2b0-311f-4ab1-9e26-e5020d81c211","Type":"ContainerStarted","Data":"4bee4f2f3d351e1d54434fa73f2dcb65ea7c603e588fdbd90e822d4fed9dc7df"} Oct 30 19:16:04.779534 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.779382 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerStarted","Data":"915a9be68ffa921c20eba306e56879cc5753c141012fe92d4178558fc33afef7"} Oct 30 19:16:04.781806 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.781343 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kxt77" event={"ID":"32f31184-b3c4-4a0e-8976-1d91195570e8","Type":"ContainerStarted","Data":"991fea18eafe223c39c5a388f7497c9ba10095ec83f11b8d4d3105d61de200e9"} Oct 30 19:16:04.811389 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:04.811331 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kxt77" podStartSLOduration=3.118263122 podStartE2EDuration="10.811315104s" totalImagesPullingTime="7.693051982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:04.810848183 +0000 UTC m=+10.692517037" watchObservedRunningTime="2026-10-30 19:16:04.811315104 +0000 UTC m=+10.692983951" Oct 30 19:16:05.574024 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.573971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:05.574181 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:05.574157 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:05.788998 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.788704 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-52x9m" event={"ID":"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc","Type":"ContainerStarted","Data":"5afba778bc05e4093034d6d65620702ede79d2bdf0960eed5705e35e51a65e8b"} Oct 30 19:16:05.791273 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.791243 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" event={"ID":"e00ba2b0-311f-4ab1-9e26-e5020d81c211","Type":"ContainerStarted","Data":"95f79aed55ab2a83e19711cb22272fde5fd1bf844c7db814ed0ded3104a78815"} Oct 30 19:16:05.794744 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.794714 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="915a9be68ffa921c20eba306e56879cc5753c141012fe92d4178558fc33afef7" exitCode=0 Oct 30 19:16:05.794861 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.794794 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerDied","Data":"915a9be68ffa921c20eba306e56879cc5753c141012fe92d4178558fc33afef7"} Oct 30 19:16:05.806612 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.806559 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-52x9m" podStartSLOduration=3.42731812 podStartE2EDuration="11.806547715s" totalImagesPullingTime="8.379229595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:05.805877525 +0000 UTC m=+11.687546373" watchObservedRunningTime="2026-10-30 19:16:05.806547715 +0000 UTC m=+11.688216562" Oct 30 19:16:05.823149 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:05.823094 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" podStartSLOduration=3.749334606 podStartE2EDuration="11.823079204s" totalImagesPullingTime="8.073744598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:05.822109989 +0000 UTC m=+11.703778837" watchObservedRunningTime="2026-10-30 19:16:05.823079204 +0000 UTC m=+11.704748052" Oct 30 19:16:06.575836 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:06.575812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:06.575930 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:06.575914 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:06.576219 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:06.576201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:06.576323 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:06.576280 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:06.800006 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:06.799918 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="c60313d7d4b38cb9cac62bef3f436f481e9a8b7525ee7ae26c519c3c80e932f9" exitCode=0 Oct 30 19:16:06.800006 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:06.799961 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerDied","Data":"c60313d7d4b38cb9cac62bef3f436f481e9a8b7525ee7ae26c519c3c80e932f9"} Oct 30 19:16:06.801476 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:06.801451 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vf6wx" event={"ID":"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7","Type":"ContainerStarted","Data":"95731e3157c9122a3e2820c76be06ed960f758f97ba14badebbe5c72f730b9ea"} Oct 30 19:16:06.835165 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:06.835113 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vf6wx" podStartSLOduration=4.829595505 podStartE2EDuration="12.835095528s" totalImagesPullingTime="8.005500023s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:06.834592259 +0000 UTC m=+12.716261107" watchObservedRunningTime="2026-10-30 19:16:06.835095528 +0000 UTC m=+12.716764378" Oct 30 19:16:07.573845 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:07.573821 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:07.573982 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:07.573959 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:07.805846 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:07.805750 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" event={"ID":"b70be47d-c307-4f47-b075-89a53633411d","Type":"ContainerStarted","Data":"e3b6dbc4e2d5d052cc15cc1fbf09bdcd9321563d3afa1a463aebcd9784707b64"} Oct 30 19:16:07.809147 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:07.809116 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="fd9ca76bf230cd3772e229f86df4a5c2a178703eafe4dff1c5572ea986073cbc" exitCode=0 Oct 30 19:16:07.809283 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:07.809211 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerDied","Data":"fd9ca76bf230cd3772e229f86df4a5c2a178703eafe4dff1c5572ea986073cbc"} Oct 30 19:16:07.825600 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:07.825558 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" podStartSLOduration=3.856336954 podStartE2EDuration="13.825545028s" totalImagesPullingTime="9.969208074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:07.825186512 +0000 UTC m=+13.706855358" watchObservedRunningTime="2026-10-30 19:16:07.825545028 +0000 UTC m=+13.707213875" Oct 30 19:16:08.573530 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:08.573492 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:08.573717 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:08.573627 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:08.574356 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:08.574337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:08.574485 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:08.574464 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:08.663687 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:08.662996 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:16:08.663687 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:08.663130 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:16:08.663687 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:08.663548 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:16:09.574254 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:09.574225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:09.574937 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:09.574363 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:09.815932 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:09.815690 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerStarted","Data":"811fc10ec793a03cab860afcdefef4a5d0a2c524f330ced4a9177a0759775729"} Oct 30 19:16:10.575182 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:10.573525 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:10.575182 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:10.573549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:10.575182 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:10.573650 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:10.575182 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:10.573780 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:10.821322 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:10.821284 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="811fc10ec793a03cab860afcdefef4a5d0a2c524f330ced4a9177a0759775729" exitCode=0 Oct 30 19:16:10.821528 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:10.821345 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerDied","Data":"811fc10ec793a03cab860afcdefef4a5d0a2c524f330ced4a9177a0759775729"} Oct 30 19:16:11.574359 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.574336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:11.574487 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:11.574470 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:11.825790 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.825719 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:16:11.826174 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826088 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" containerID="fadeb12c6aa6b5ae9923a81221dbee4bbb308dac755b181d2393c0e5d027faef" exitCode=1 Oct 30 19:16:11.826174 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826164 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"aeeb4f00993400dc4415b00cd8692ea7708cc560cd268dcbec5131a214f44a55"} Oct 30 19:16:11.826273 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826257 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"d2bf80081ea845c8f7a5d8e2a830bbd82f291bd6d5abb147a9f882df76c0afc8"} Oct 30 19:16:11.826314 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826293 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"2f3492021047d50e8c92d9139c00b28162875c66fe8cf5985e5dd49d96a52755"} Oct 30 19:16:11.826343 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826321 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"0fcdcee285cfbc6f6246eff64c8a84ff9de70441c4b5ecf8f47d8b4cbf8aee75"} Oct 30 19:16:11.826391 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826361 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerDied","Data":"fadeb12c6aa6b5ae9923a81221dbee4bbb308dac755b181d2393c0e5d027faef"} Oct 30 19:16:11.826450 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.826402 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"cc60d227fdb0303f896f2c340d94d657e4079628e65d2f6e1c26dfac64181593"} Oct 30 19:16:11.829248 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.829231 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="22f8592a5f4c026ae3e51cca4ca9f66fdee9dc8a25b9109ef558133f58c3d0ca" exitCode=0 Oct 30 19:16:11.829296 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:11.829269 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerDied","Data":"22f8592a5f4c026ae3e51cca4ca9f66fdee9dc8a25b9109ef558133f58c3d0ca"} Oct 30 19:16:12.353620 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.353590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:12.353758 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.353624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:12.353758 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.353705 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:12.353758 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.353718 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:12.353758 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.353749 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:28.353737043 +0000 UTC m=+34.235405868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:12.353907 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.353766 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:16:28.3537544 +0000 UTC m=+34.235423249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:12.454239 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.454213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:12.454359 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.454334 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:12.454359 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.454349 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:12.454359 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.454358 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:12.454488 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.454400 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:28.454384959 +0000 UTC m=+34.336053798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:12.574039 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.574012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:12.574176 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.574116 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:12.574233 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.574180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:12.574301 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:12.574284 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:12.834024 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.833948 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" event={"ID":"0e6ea1e8-21c6-48fb-beb1-7246750874a4","Type":"ContainerStarted","Data":"8361fc9b25faa694c6c454e5cced32edd9553e44a8c4be17bcf595986c47cd98"} Oct 30 19:16:12.856818 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:12.856776 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gh6sj" podStartSLOduration=8.363812169 podStartE2EDuration="18.856763035s" totalImagesPullingTime="10.492950866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:12.85635225 +0000 UTC m=+18.738021099" watchObservedRunningTime="2026-10-30 19:16:12.856763035 +0000 UTC m=+18.738431881" Oct 30 19:16:13.574284 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:13.574254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:13.574440 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:13.574354 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:13.839277 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:13.839205 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:16:13.839662 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:13.839578 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"566e649480fc0b3747b976b7faf57436f3d35260a592352568439a2cc4efdc16"} Oct 30 19:16:14.574053 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:14.574021 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:14.574234 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:14.574119 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:14.574234 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:14.574180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:14.574351 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:14.574267 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:15.573802 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.573764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:15.574260 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:15.573879 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:15.847577 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.847511 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:16:15.847909 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.847883 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"34a3c124265ca5d077b5edbc5199399e9b9ae72d3a195e9f7db228adde7a6758"} Oct 30 19:16:15.848271 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.848246 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:16:15.848373 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.848329 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:16:15.848441 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.848376 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:16:15.848441 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.848400 2575 scope.go:122] "RemoveContainer" containerID="fadeb12c6aa6b5ae9923a81221dbee4bbb308dac755b181d2393c0e5d027faef" Oct 30 19:16:15.865149 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.864860 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:16:15.866948 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:15.866926 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:16:16.573740 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:16.573708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:16.573740 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:16.573734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:16.574782 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:16.573804 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:16.574782 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:16.573873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:16.853444 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:16.853369 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:16:16.853756 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:16.853734 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" event={"ID":"a3ce0b21-0196-4d7c-9b36-99ec47e7d057","Type":"ContainerStarted","Data":"9fa398f46986491f670e9cf0483fb6e94ba2c6349f1e2cf48e7b68703ecf2dfd"} Oct 30 19:16:16.888015 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:16.887976 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" podStartSLOduration=8.589597746 podStartE2EDuration="22.88796498s" totalImagesPullingTime="14.298367234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:16.886495807 +0000 UTC m=+22.768164659" watchObservedRunningTime="2026-10-30 19:16:16.88796498 +0000 UTC m=+22.769633828" Oct 30 19:16:17.474503 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:17.474398 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-468t5"] Oct 30 19:16:17.474710 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:17.474550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:17.474710 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:17.474639 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:17.479580 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:17.479552 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9hzhx"] Oct 30 19:16:17.479695 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:17.479675 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:17.479858 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:17.479792 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:17.480765 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:17.480744 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6wtd"] Oct 30 19:16:17.480865 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:17.480851 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:17.480965 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:17.480950 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:18.008435 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:18.008395 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxt77_32f31184-b3c4-4a0e-8976-1d91195570e8/dns-node-resolver/0.log" Oct 30 19:16:18.990264 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:18.990239 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4wfl7_a00a389a-2d47-4c51-9e90-85e9ffe3c9d6/node-ca/0.log" Oct 30 19:16:19.573554 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:19.573524 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:19.573943 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:19.573619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:19.573943 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:19.573627 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:19.573943 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:19.573703 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:19.573943 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:19.573743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:19.573943 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:19.573795 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:21.574300 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:21.574265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:21.574727 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:21.574313 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:21.574727 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:21.574399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:21.574727 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:21.574407 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:21.574727 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:21.574490 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:21.574727 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:21.574574 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:23.574092 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:23.574011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:23.574536 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:23.574126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:23.574536 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:23.574128 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:23.574536 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:23.574143 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:23.574536 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:23.574247 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:23.574536 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:23.574337 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:25.573663 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:25.573627 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:25.573663 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:25.573650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:25.574074 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:25.573730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:25.574074 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:25.573741 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:25.574074 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:25.573789 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:25.574074 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:25.573883 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:27.573365 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:27.573303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:27.573740 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:27.573303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:27.573740 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:27.573395 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:27.573740 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:27.573303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:27.573740 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:27.573485 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:27.573740 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:27.573556 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:28.369005 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:28.368975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:28.369164 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:28.369042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:28.369164 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.369116 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:28.369241 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.369169 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret podName:da8143dd-2ae4-4932-91b9-9215cf9cd469 nodeName:}" failed. No retries permitted until 2026-10-30 19:17:00.369154911 +0000 UTC m=+66.250823735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret") pod "global-pull-secret-syncer-f6wtd" (UID: "da8143dd-2ae4-4932-91b9-9215cf9cd469") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:16:28.369241 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.369116 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:28.369241 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.369231 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs podName:3b0c721a-72d7-44ce-af65-457dc171bfca nodeName:}" failed. No retries permitted until 2026-10-30 19:17:00.369218257 +0000 UTC m=+66.250887096 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs") pod "network-metrics-daemon-468t5" (UID: "3b0c721a-72d7-44ce-af65-457dc171bfca") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:16:28.469699 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:28.469672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:28.469844 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.469781 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:16:28.469844 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.469796 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:16:28.469844 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.469804 2575 projected.go:196] Error preparing data for projected volume kube-api-access-lpxvl for pod openshift-network-diagnostics/network-check-target-9hzhx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:28.469844 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:28.469845 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl podName:18670897-26cd-499f-900b-8d02e8ded5d6 nodeName:}" failed. No retries permitted until 2026-10-30 19:17:00.469833404 +0000 UTC m=+66.351502229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lpxvl" (UniqueName: "kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl") pod "network-check-target-9hzhx" (UID: "18670897-26cd-499f-900b-8d02e8ded5d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:16:29.573917 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:29.573882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:29.574303 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:29.573937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:29.574303 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:29.574031 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:29.574303 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:29.574081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:29.574303 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:29.574156 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:29.574303 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:29.574218 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:31.574226 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:31.574190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:31.574226 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:31.574213 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:31.574665 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:31.574298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:31.574665 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:31.574319 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:31.574665 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:31.574369 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:31.574665 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:31.574458 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:33.573945 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:33.573915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:33.574448 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:33.574013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:33.574448 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:33.574020 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:33.574448 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:33.574041 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:33.574448 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:33.574122 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:33.574448 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:33.574184 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:35.573720 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:35.573686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:35.574092 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:35.573688 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:35.574092 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:35.573789 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wtd" podUID="da8143dd-2ae4-4932-91b9-9215cf9cd469" Oct 30 19:16:35.574092 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:35.573689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:35.574092 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:35.573868 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-468t5" podUID="3b0c721a-72d7-44ce-af65-457dc171bfca" Oct 30 19:16:35.574092 ip-10-0-140-127 kubenswrapper[2575]: E1030 19:16:35.573938 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hzhx" podUID="18670897-26cd-499f-900b-8d02e8ded5d6" Oct 30 19:16:36.902657 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.902591 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeReady" Oct 30 19:16:36.902998 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.902717 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:16:36.946644 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.946613 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4wmdq"] Oct 30 19:16:36.948645 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.948630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:36.949980 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.949959 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kkdfr"] Oct 30 19:16:36.951664 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.951650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:36.954501 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.954301 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:16:36.954501 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.954322 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:16:36.954501 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.954381 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:16:36.954501 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.954464 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bfjrl\"" Oct 30 19:16:36.954793 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.954778 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:16:36.955341 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.955328 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:16:36.955592 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.955577 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:16:36.955668 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.955592 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:16:36.956327 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.956069 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7b7kv\"" Oct 30 19:16:36.962728 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.962708 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wmdq"] Oct 30 19:16:36.964623 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.964602 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkdfr"] Oct 30 19:16:36.972209 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.972189 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5vdt7"] Oct 30 19:16:36.973870 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.973857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:36.984352 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:36.984330 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5vdt7"] Oct 30 19:16:37.032171 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032147 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eda631-51b9-4c18-ad28-99ebb93e0db6-config-volume\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.032289 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69df0eef-ee4f-4135-ab6c-01e806705d4a-data-volume\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.032289 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.032289 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ng9\" (UniqueName: \"kubernetes.io/projected/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-api-access-z2ng9\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.032289 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvcx\" (UniqueName: \"kubernetes.io/projected/d0eda631-51b9-4c18-ad28-99ebb93e0db6-kube-api-access-bsvcx\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.032458 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69df0eef-ee4f-4135-ab6c-01e806705d4a-crio-socket\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.032458 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69df0eef-ee4f-4135-ab6c-01e806705d4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.032458 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0eda631-51b9-4c18-ad28-99ebb93e0db6-metrics-tls\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.032558 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8wbv\" (UniqueName: \"kubernetes.io/projected/68413742-d011-4c65-8ce8-2cf80d43ac1a-kube-api-access-k8wbv\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.032558 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0eda631-51b9-4c18-ad28-99ebb93e0db6-tmp-dir\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.032558 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.032547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68413742-d011-4c65-8ce8-2cf80d43ac1a-cert\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.053798 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.053782 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:16:37.053854 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.053781 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:16:37.053854 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.053785 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfmhh\"" Oct 30 19:16:37.133142 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8wbv\" (UniqueName: \"kubernetes.io/projected/68413742-d011-4c65-8ce8-2cf80d43ac1a-kube-api-access-k8wbv\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.133142 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0eda631-51b9-4c18-ad28-99ebb93e0db6-tmp-dir\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.133299 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68413742-d011-4c65-8ce8-2cf80d43ac1a-cert\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.133299 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eda631-51b9-4c18-ad28-99ebb93e0db6-config-volume\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.133373 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69df0eef-ee4f-4135-ab6c-01e806705d4a-data-volume\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133457 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133457 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ng9\" (UniqueName: \"kubernetes.io/projected/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-api-access-z2ng9\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133574 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvcx\" (UniqueName: \"kubernetes.io/projected/d0eda631-51b9-4c18-ad28-99ebb93e0db6-kube-api-access-bsvcx\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.133574 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69df0eef-ee4f-4135-ab6c-01e806705d4a-crio-socket\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133574 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69df0eef-ee4f-4135-ab6c-01e806705d4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133574 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0eda631-51b9-4c18-ad28-99ebb93e0db6-metrics-tls\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.133779 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0eda631-51b9-4c18-ad28-99ebb93e0db6-tmp-dir\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.133779 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133693 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69df0eef-ee4f-4135-ab6c-01e806705d4a-data-volume\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133878 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133801 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69df0eef-ee4f-4135-ab6c-01e806705d4a-crio-socket\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.133941 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.133893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eda631-51b9-4c18-ad28-99ebb93e0db6-config-volume\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.134035 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.134015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.137196 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.137180 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0eda631-51b9-4c18-ad28-99ebb93e0db6-metrics-tls\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.137286 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.137232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69df0eef-ee4f-4135-ab6c-01e806705d4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.137357 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.137342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68413742-d011-4c65-8ce8-2cf80d43ac1a-cert\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.141524 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.141500 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ng9\" (UniqueName: \"kubernetes.io/projected/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-api-access-z2ng9\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.141524 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.141521 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvcx\" (UniqueName: \"kubernetes.io/projected/d0eda631-51b9-4c18-ad28-99ebb93e0db6-kube-api-access-bsvcx\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.141753 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.141730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8wbv\" (UniqueName: \"kubernetes.io/projected/68413742-d011-4c65-8ce8-2cf80d43ac1a-kube-api-access-k8wbv\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.260246 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.260186 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:16:37.268526 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.268506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:16:37.283483 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.283458 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:37.539290 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.539216 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wmdq"] Oct 30 19:16:37.545219 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.545179 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkdfr"] Oct 30 19:16:37.548587 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:16:37.548563 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69df0eef_ee4f_4135_ab6c_01e806705d4a.slice/crio-7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6 WatchSource:0}: Error finding container 7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6: Status 404 returned error can't find the container with id 7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6 Oct 30 19:16:37.562319 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.562299 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5vdt7"] Oct 30 19:16:37.564447 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:16:37.564403 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0eda631_51b9_4c18_ad28_99ebb93e0db6.slice/crio-48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09 WatchSource:0}: Error finding container 48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09: Status 404 returned error can't find the container with id 48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09 Oct 30 19:16:37.574189 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.574171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:16:37.574269 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.574197 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:16:37.574355 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.574333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:16:37.581184 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.581077 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:16:37.581277 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.581220 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:16:37.581959 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.581938 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:16:37.582039 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.582011 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\"" Oct 30 19:16:37.582039 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.582011 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:16:37.582545 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.582533 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\"" Oct 30 19:16:37.582613 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.582579 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\"" Oct 30 19:16:37.897822 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.897784 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5vdt7" event={"ID":"d0eda631-51b9-4c18-ad28-99ebb93e0db6","Type":"ContainerStarted","Data":"48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09"} Oct 30 19:16:37.899323 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.899297 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkdfr" event={"ID":"69df0eef-ee4f-4135-ab6c-01e806705d4a","Type":"ContainerStarted","Data":"9d5df9980e319e9fc6bf18fc26bd5bed9fb20aec0d893affbcf5b4f1291aa0a0"} Oct 30 19:16:37.899456 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.899352 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkdfr" event={"ID":"69df0eef-ee4f-4135-ab6c-01e806705d4a","Type":"ContainerStarted","Data":"7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6"} Oct 30 19:16:37.900328 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:37.900309 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wmdq" event={"ID":"68413742-d011-4c65-8ce8-2cf80d43ac1a","Type":"ContainerStarted","Data":"d255784a66cba04a1260b7894c80be5f96d82d7e15857fe994cc010b04af5e9a"} Oct 30 19:16:38.599011 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:38.598928 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxt77_32f31184-b3c4-4a0e-8976-1d91195570e8/dns-node-resolver/0.log" Oct 30 19:16:38.906038 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:38.905906 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkdfr" event={"ID":"69df0eef-ee4f-4135-ab6c-01e806705d4a","Type":"ContainerStarted","Data":"d8cf80e1f3addcc360aa1c361df58d44bf9031d893c359240fef6a4edb5af0da"} Oct 30 19:16:39.173343 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.173251 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7mc52"] Oct 30 19:16:39.175631 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.175605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.182591 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.182568 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:16:39.185557 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.185538 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:16:39.187444 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.186341 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 19:16:39.187892 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.187841 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l9rvz\"" Oct 30 19:16:39.188070 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.188049 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 19:16:39.188135 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.188127 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7mc52"] Oct 30 19:16:39.251598 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.251549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.251781 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.251666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxt7\" (UniqueName: \"kubernetes.io/projected/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-kube-api-access-9qxt7\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.251855 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.251799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.251855 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.251848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.284521 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.284471 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:16:39.354759 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.353146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.354759 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.353205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxt7\" (UniqueName: \"kubernetes.io/projected/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-kube-api-access-9qxt7\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.354759 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.353278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.354759 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.353320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.354759 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.354156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.359454 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.357700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.359454 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.358605 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.362372 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.362334 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxt7\" (UniqueName: \"kubernetes.io/projected/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-kube-api-access-9qxt7\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.493931 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.493672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:16:39.779667 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.779617 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7mc52"] Oct 30 19:16:39.783240 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:16:39.783217 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752b9e9a_aba1_4aee_8e54_83aa3ea6c854.slice/crio-9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945 WatchSource:0}: Error finding container 9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945: Status 404 returned error can't find the container with id 9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945 Oct 30 19:16:39.798259 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.798181 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4wfl7_a00a389a-2d47-4c51-9e90-85e9ffe3c9d6/node-ca/0.log" Oct 30 19:16:39.911574 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.911535 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5vdt7" event={"ID":"d0eda631-51b9-4c18-ad28-99ebb93e0db6","Type":"ContainerStarted","Data":"1119539a02fb4d8bf828309a6b6acfc30cbce19965117d42d46c7bbb73fd60e3"} Oct 30 19:16:39.911761 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.911605 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5vdt7" event={"ID":"d0eda631-51b9-4c18-ad28-99ebb93e0db6","Type":"ContainerStarted","Data":"c0765ea112aace2e496afe0fb9274e6a837aac8f2d92a201546e82d60ba72754"} Oct 30 19:16:39.911761 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.911653 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:39.913001 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:39.912976 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" event={"ID":"752b9e9a-aba1-4aee-8e54-83aa3ea6c854","Type":"ContainerStarted","Data":"9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945"} Oct 30 19:16:40.917404 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:40.917358 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wmdq" event={"ID":"68413742-d011-4c65-8ce8-2cf80d43ac1a","Type":"ContainerStarted","Data":"7d022b3a593b1a9ccf8c4cb8a70e2e63547f1e513222b1fbd656cab07ccb4bcd"} Oct 30 19:16:40.919918 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:40.919891 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkdfr" event={"ID":"69df0eef-ee4f-4135-ab6c-01e806705d4a","Type":"ContainerStarted","Data":"5a71e497a04dcc6ff270762c3d9738c75c8b0d5c9454805bd744ec74bf4808dc"} Oct 30 19:16:40.936375 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:40.936326 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4wmdq" podStartSLOduration=2.124519842 podStartE2EDuration="4.936312945s" totalImagesPullingTime="2.811793103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:40.933935841 +0000 UTC m=+46.815604691" watchObservedRunningTime="2026-10-30 19:16:40.936312945 +0000 UTC m=+46.817981828" Oct 30 19:16:40.936526 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:40.936450 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5vdt7" podStartSLOduration=2.940178681 podStartE2EDuration="4.93644096s" totalImagesPullingTime="1.996262279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:39.930269214 +0000 UTC m=+45.811938064" watchObservedRunningTime="2026-10-30 19:16:40.93644096 +0000 UTC m=+46.818109801" Oct 30 19:16:40.961940 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:40.961894 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kkdfr" podStartSLOduration=2.692872171 podStartE2EDuration="4.961877128s" totalImagesPullingTime="2.269004957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:40.960247337 +0000 UTC m=+46.841916185" watchObservedRunningTime="2026-10-30 19:16:40.961877128 +0000 UTC m=+46.843545977" Oct 30 19:16:41.925168 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:41.925130 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" event={"ID":"752b9e9a-aba1-4aee-8e54-83aa3ea6c854","Type":"ContainerStarted","Data":"bc991215bf3342c399a0d5e240965f609b02e37f17434b2ac3a8ebd3dca28be9"} Oct 30 19:16:41.925619 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:41.925183 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" event={"ID":"752b9e9a-aba1-4aee-8e54-83aa3ea6c854","Type":"ContainerStarted","Data":"2357a2558c0dc3e5056459ca83e00b822adaa9efb6dde8dfa99e8c827c357840"} Oct 30 19:16:41.942948 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:41.942905 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" podStartSLOduration=1.63694492 podStartE2EDuration="2.94289219s" totalImagesPullingTime="1.30594727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:41.941227968 +0000 UTC m=+47.822896816" watchObservedRunningTime="2026-10-30 19:16:41.94289219 +0000 UTC m=+47.824561037" Oct 30 19:16:43.580351 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.580306 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pfp6b"] Oct 30 19:16:43.583692 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.583650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.603145 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.603122 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:16:43.603145 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.603128 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:16:43.603473 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.603455 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bc6qp\"" Oct 30 19:16:43.649922 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.649897 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:16:43.684270 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-wtmp\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-tls\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684377 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684469 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-root\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684500 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-metrics-client-ca\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684500 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-textfile\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684558 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684558 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5st4\" (UniqueName: \"kubernetes.io/projected/ff4585e9-d9f1-4736-aab7-485d22d6888c-kube-api-access-g5st4\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.684617 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.684603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-sys\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.784817 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-tls\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.784924 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.784924 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-root\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.784924 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-metrics-client-ca\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.784924 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-textfile\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.784924 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785174 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784939 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-root\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785174 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.784993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5st4\" (UniqueName: \"kubernetes.io/projected/ff4585e9-d9f1-4736-aab7-485d22d6888c-kube-api-access-g5st4\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785174 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-sys\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785174 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-wtmp\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785352 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-wtmp\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785352 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-textfile\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785352 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785320 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-sys\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785473 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.785555 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.785535 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-metrics-client-ca\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.787311 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.787288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-tls\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.787394 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.787335 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.793617 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.793600 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5st4\" (UniqueName: \"kubernetes.io/projected/ff4585e9-d9f1-4736-aab7-485d22d6888c-kube-api-access-g5st4\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.895106 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.895040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:16:43.905244 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:16:43.905220 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4585e9_d9f1_4736_aab7_485d22d6888c.slice/crio-ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3 WatchSource:0}: Error finding container ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3: Status 404 returned error can't find the container with id ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3 Oct 30 19:16:43.932562 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:43.932537 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfp6b" event={"ID":"ff4585e9-d9f1-4736-aab7-485d22d6888c","Type":"ContainerStarted","Data":"ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3"} Oct 30 19:16:45.650269 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.650234 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7646648d5d-jd8l2"] Oct 30 19:16:45.653763 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.653739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.657518 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.657496 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:16:45.657733 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.657502 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:16:45.658254 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.658227 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-29o98lk1qr308\"" Oct 30 19:16:45.658367 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.658275 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:16:45.658367 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.658284 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:16:45.658367 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.658363 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:16:45.658514 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.658385 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2lpdx\"" Oct 30 19:16:45.665205 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.665183 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7646648d5d-jd8l2"] Oct 30 19:16:45.700820 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.700798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.700934 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.700827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.700934 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.700845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2l6\" (UniqueName: \"kubernetes.io/projected/69ea1b29-9274-49ce-a4b1-81a8aa191296-kube-api-access-wr2l6\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.700934 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.700863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.700934 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.700918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea1b29-9274-49ce-a4b1-81a8aa191296-tmp\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.701054 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.700952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.701094 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.701064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-grpc-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.701128 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.701107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ea1b29-9274-49ce-a4b1-81a8aa191296-metrics-client-ca\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.701158 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.701135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802431 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802550 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea1b29-9274-49ce-a4b1-81a8aa191296-tmp\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802550 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802550 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-grpc-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802550 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ea1b29-9274-49ce-a4b1-81a8aa191296-metrics-client-ca\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802550 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802766 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802766 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802766 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2l6\" (UniqueName: \"kubernetes.io/projected/69ea1b29-9274-49ce-a4b1-81a8aa191296-kube-api-access-wr2l6\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.802940 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.802900 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea1b29-9274-49ce-a4b1-81a8aa191296-tmp\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.803394 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.803310 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ea1b29-9274-49ce-a4b1-81a8aa191296-metrics-client-ca\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.805272 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.805234 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.805514 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.805491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-grpc-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.805685 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.805659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.805957 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.805936 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.806057 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.806033 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.806096 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.806038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.810811 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.810794 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2l6\" (UniqueName: \"kubernetes.io/projected/69ea1b29-9274-49ce-a4b1-81a8aa191296-kube-api-access-wr2l6\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:45.941950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.941874 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff4585e9-d9f1-4736-aab7-485d22d6888c" containerID="e1b34dbed9172ecddcd6374c5f58a8c3f81057b0618cfcaf22842d6e672e8d38" exitCode=0 Oct 30 19:16:45.941950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.941940 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfp6b" event={"ID":"ff4585e9-d9f1-4736-aab7-485d22d6888c","Type":"ContainerDied","Data":"e1b34dbed9172ecddcd6374c5f58a8c3f81057b0618cfcaf22842d6e672e8d38"} Oct 30 19:16:45.981093 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:45.981069 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:46.259475 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:46.259450 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7646648d5d-jd8l2"] Oct 30 19:16:46.261846 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:16:46.261816 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ea1b29_9274_49ce_a4b1_81a8aa191296.slice/crio-19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56 WatchSource:0}: Error finding container 19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56: Status 404 returned error can't find the container with id 19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56 Oct 30 19:16:46.948819 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:46.948778 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfp6b" event={"ID":"ff4585e9-d9f1-4736-aab7-485d22d6888c","Type":"ContainerStarted","Data":"a353c54f51a325bb3f52c5e47314c0270f73a382adb491881132534ea2e71027"} Oct 30 19:16:46.949310 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:46.948849 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfp6b" event={"ID":"ff4585e9-d9f1-4736-aab7-485d22d6888c","Type":"ContainerStarted","Data":"76824c461b257a6d64023c34276253f5f13c383661a0afbaae222d677f15160b"} Oct 30 19:16:46.950311 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:46.950279 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56"} Oct 30 19:16:46.969612 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:46.969555 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pfp6b" podStartSLOduration=2.882904976 podStartE2EDuration="3.969538126s" totalImagesPullingTime="1.010079902s" totalInitContainerRuntime="76.553248ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:46.96875403 +0000 UTC m=+52.850422877" watchObservedRunningTime="2026-10-30 19:16:46.969538126 +0000 UTC m=+52.851206975" Oct 30 19:16:47.867520 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:47.867488 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:16:48.957580 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:48.957546 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"7ab9af4bc005abf5cecb745f8ae4c25e55e9891e4dbb11340d92a7ddccdaf2b0"} Oct 30 19:16:48.957938 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:48.957612 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"154012554bf5d5c10dad076384a5f83cbeb488088d48b8b7361cf26a25cce756"} Oct 30 19:16:48.957938 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:48.957637 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"11635ebd0b528468ffffd1645fc50e34d6ea04701e6f41fc4c4b5ec850e387a8"} Oct 30 19:16:49.792232 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.792196 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:16:49.796197 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.796173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.802004 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.801914 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:16:49.802004 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.801928 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:16:49.802004 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.801947 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:16:49.802004 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.801982 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rklsm\"" Oct 30 19:16:49.802004 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.801991 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:16:49.802343 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.802028 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:16:49.802343 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.802028 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:16:49.802343 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.802075 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:16:49.802343 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.802211 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rd2gbba2224\"" Oct 30 19:16:49.803111 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.803092 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:16:49.803111 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.803095 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:16:49.806950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.806359 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:16:49.806950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.806365 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:16:49.806950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.806463 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:16:49.806950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.806468 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:16:49.806950 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.806497 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:16:49.808836 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.808818 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:16:49.818123 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.817457 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:16:49.837267 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837267 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837462 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837462 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837462 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837621 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837621 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837621 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837769 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837769 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837838 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837838 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837909 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837909 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837968 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.837968 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.838034 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.838034 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.837999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.838152 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.838039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.838152 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.838076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.922909 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.922884 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5vdt7" Oct 30 19:16:49.939057 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.939014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.939057 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.939058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.939253 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.939086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.939253 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.939113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.939253 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.939146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.939708 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.939678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.940904 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.940044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.940904 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.940095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.940904 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.940311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.941392 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.941369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.941820 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.941801 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.942483 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.942637 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942622 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.942785 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.942875 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942845 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.942978 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.942978 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943083 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.942979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943083 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943083 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943083 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943264 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943264 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943264 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943551 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.943638 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.944015 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.943991 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.944111 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.944065 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.944769 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.944643 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.945095 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.945069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.946016 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.945636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.946507 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.946270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.947893 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.947501 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.948203 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.948140 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.950639 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.948619 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.950639 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.948766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.950639 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.950138 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.952069 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.951980 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.954457 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.954387 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.956016 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.955995 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:49.966012 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.965986 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"eb10f2436ccb5ae669eca747347efdca1fc59819a2b57ed18e3e4047350aafde"} Oct 30 19:16:49.966331 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.966096 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"48c9507aa830d8c02f1b251a4c0868cf3eb99d7222763b0e71c017ee24fbfd82"} Oct 30 19:16:49.966331 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.966142 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" event={"ID":"69ea1b29-9274-49ce-a4b1-81a8aa191296","Type":"ContainerStarted","Data":"8d97bac498f5a893c5c3f22f7624201b3e6e70540c84843438bf037e8905f8e2"} Oct 30 19:16:49.966331 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.966214 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:49.990787 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:49.990734 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" podStartSLOduration=2.420748503 podStartE2EDuration="4.990715348s" totalImagesPullingTime="2.569966845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:49.989836065 +0000 UTC m=+55.871504912" watchObservedRunningTime="2026-10-30 19:16:49.990715348 +0000 UTC m=+55.872384196" Oct 30 19:16:50.127724 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:50.127686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:16:50.414866 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:50.414789 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:16:50.418293 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:16:50.418266 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7951e5f1_5cb5_4185_836d_599ac3d2223f.slice/crio-2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355 WatchSource:0}: Error finding container 2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355: Status 404 returned error can't find the container with id 2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355 Oct 30 19:16:50.971062 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:50.971028 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355"} Oct 30 19:16:51.975987 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:51.975944 2575 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" exitCode=0 Oct 30 19:16:51.976459 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:51.976038 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} Oct 30 19:16:55.978216 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.978187 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:16:55.993186 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.993161 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} Oct 30 19:16:55.993329 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.993215 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} Oct 30 19:16:55.993329 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.993240 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} Oct 30 19:16:55.993329 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.993264 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} Oct 30 19:16:55.993329 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.993287 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} Oct 30 19:16:55.993329 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:55.993308 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerStarted","Data":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} Oct 30 19:16:56.040176 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:16:56.040126 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.012087409 podStartE2EDuration="7.040109145s" totalImagesPullingTime="4.012273733s" totalInitContainerRuntime="15.748003ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:56.039945042 +0000 UTC m=+61.921613893" watchObservedRunningTime="2026-10-30 19:16:56.040109145 +0000 UTC m=+61.921777994" Oct 30 19:17:00.128445 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.128392 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:00.449373 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.449284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:00.449373 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.449343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:00.452019 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.451999 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:17:00.452878 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.452862 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:17:00.462252 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.462221 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:00.462365 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.462348 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:00.549862 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.549828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:00.552462 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.552445 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:17:00.563116 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.563098 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:17:00.573583 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.573561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:00.702381 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.702299 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\"" Oct 30 19:17:00.706762 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.706742 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\"" Oct 30 19:17:00.709907 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.709890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:00.710281 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.710268 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\"" Oct 30 19:17:00.714802 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.714787 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:00.718272 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.718253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:00.991368 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.991327 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-468t5"] Oct 30 19:17:00.994851 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.994831 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9hzhx"] Oct 30 19:17:00.997922 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:00.997905 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6wtd"] Oct 30 19:17:00.998553 ip-10-0-140-127 kubenswrapper[2575]: W1030 19:17:00.998531 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18670897_26cd_499f_900b_8d02e8ded5d6.slice/crio-1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425 WatchSource:0}: Error finding container 1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425: Status 404 returned error can't find the container with id 1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425 Oct 30 19:17:01.022607 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:01.022575 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6wtd" event={"ID":"da8143dd-2ae4-4932-91b9-9215cf9cd469","Type":"ContainerStarted","Data":"f0368674a6b971596f3c3c40b063b5be6d75bfd87b75ef464142f963843c486e"} Oct 30 19:17:01.023630 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:01.023609 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-468t5" event={"ID":"3b0c721a-72d7-44ce-af65-457dc171bfca","Type":"ContainerStarted","Data":"d9ea2100d1af2c9de15a75a20df278853baa10d6daf6966471c3081eb9322694"} Oct 30 19:17:01.028449 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:01.025044 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9hzhx" event={"ID":"18670897-26cd-499f-900b-8d02e8ded5d6","Type":"ContainerStarted","Data":"1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425"} Oct 30 19:17:03.037540 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:03.037502 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-468t5" event={"ID":"3b0c721a-72d7-44ce-af65-457dc171bfca","Type":"ContainerStarted","Data":"b6e53e3345fb464dc6100e3f1db386e5811f9f3d82bdcb60262bd1e960540415"} Oct 30 19:17:03.037990 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:03.037571 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-468t5" event={"ID":"3b0c721a-72d7-44ce-af65-457dc171bfca","Type":"ContainerStarted","Data":"0b7729cef0c713c33083d8b497e6587c2beaf6dbe8c184a76b045466683c2af5"} Oct 30 19:17:03.055621 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:03.055571 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-468t5" podStartSLOduration=67.887633975 podStartE2EDuration="1m9.055558434s" totalImagesPullingTime="1.167924459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:03.055098808 +0000 UTC m=+68.936767655" watchObservedRunningTime="2026-10-30 19:17:03.055558434 +0000 UTC m=+68.937227280" Oct 30 19:17:05.048742 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:05.048698 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9hzhx" event={"ID":"18670897-26cd-499f-900b-8d02e8ded5d6","Type":"ContainerStarted","Data":"d6539558a5abd318c9d13deb32115cd814976627ddd635a44b8c1415ac257bfb"} Oct 30 19:17:05.049115 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:05.048828 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:05.064576 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:05.064518 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9hzhx" podStartSLOduration=67.8685699 podStartE2EDuration="1m11.064501771s" totalImagesPullingTime="3.195931871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:05.064076783 +0000 UTC m=+70.945745631" watchObservedRunningTime="2026-10-30 19:17:05.064501771 +0000 UTC m=+70.946170620" Oct 30 19:17:05.920977 ip-10-0-140-127 kubenswrapper[2575]: I1030 19:17:05.920871 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:17:05.921053 ip-10-0-140-127 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:17:05.931861 ip-10-0-140-127 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:17:05.932026 ip-10-0-140-127 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:17:05.932074 ip-10-0-140-127 systemd[1]: kubelet.service: Consumed 4.008s CPU time. Oct 30 19:17:05.943956 ip-10-0-140-127 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:17:06.006026 ip-10-0-140-127 kubenswrapper[6511]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:17:06.006026 ip-10-0-140-127 kubenswrapper[6511]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:17:06.006026 ip-10-0-140-127 kubenswrapper[6511]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:17:06.006026 ip-10-0-140-127 kubenswrapper[6511]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:17:06.008043 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.007970 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.008043 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008043 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008048 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008051 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008055 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008057 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008060 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008063 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008066 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008068 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008071 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008077 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008080 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008083 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008085 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008088 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008091 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008093 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008096 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008098 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008101 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.008119 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008103 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008106 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008108 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008111 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008113 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008116 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008118 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008120 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008124 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008126 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008129 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008132 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008135 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008137 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008140 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008142 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008145 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008147 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008150 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.008632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008152 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008154 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008157 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008159 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008162 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008166 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008171 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008174 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008177 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008180 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008183 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008185 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008189 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008192 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008195 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008198 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008201 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008204 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008206 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.009084 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008209 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008211 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008214 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008216 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008219 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008222 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008224 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008227 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008229 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008232 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008234 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008237 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008240 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008243 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008245 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008248 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008250 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008253 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008256 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008258 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.009564 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008261 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008263 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008266 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008269 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008271 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008274 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008277 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008279 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008282 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008284 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008287 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008290 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008292 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008296 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008298 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008301 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008303 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008305 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008308 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.010048 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008310 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008313 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008315 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008317 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008320 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008322 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008325 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.008328 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009070 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009077 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009080 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009083 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009086 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009089 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009094 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009097 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009101 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009105 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009109 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009111 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.010524 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009114 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009117 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009119 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009122 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009124 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009127 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009129 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009132 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009134 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009137 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009140 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009143 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009145 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009148 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009151 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009153 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009156 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009158 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009161 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009163 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.011037 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009166 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009168 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009170 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009173 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009175 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009180 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009183 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009186 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009188 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009191 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009193 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009196 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009199 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009202 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009204 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009207 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009209 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009212 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009214 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009217 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.011574 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009219 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009222 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009224 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009230 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009232 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009235 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009238 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009240 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009243 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009245 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009247 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009250 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009252 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009255 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009257 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009260 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009262 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009265 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009268 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.012100 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009271 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009273 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009276 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009278 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009281 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009284 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009287 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009289 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009292 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009294 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009297 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009299 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009302 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009304 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009307 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009309 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009313 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009316 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009318 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.012587 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009321 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009323 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009326 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009329 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009331 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009334 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009336 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009339 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009341 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009344 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009347 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009349 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009352 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009355 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009358 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009360 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009861 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009868 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009871 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.013054 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009874 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009877 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009880 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009883 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009886 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009888 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009891 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009893 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009896 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009898 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009903 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009906 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009908 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009911 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009913 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009916 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009918 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009921 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009923 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009926 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.013616 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009928 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009931 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009935 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009938 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009941 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009944 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009948 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009951 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009954 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009957 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009959 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009962 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009964 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009967 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009969 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009972 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009974 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009977 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009979 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.014538 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009982 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009984 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009987 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009990 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009993 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009995 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.009998 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010000 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010002 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010005 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010007 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010010 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010012 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010015 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010017 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010020 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010022 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010026 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010029 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010031 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.015364 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010034 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010037 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010039 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010041 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010044 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010047 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010049 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010052 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010055 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010058 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010060 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010063 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010065 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010068 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010070 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010073 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010076 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010078 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010081 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010083 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.016276 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010086 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010088 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010090 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010093 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010095 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010097 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010100 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010102 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010105 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010111 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010113 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010116 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010118 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010121 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010124 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010126 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010129 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010132 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010135 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010137 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.017334 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010139 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010142 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010144 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010147 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010236 6511 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010243 6511 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010248 6511 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010253 6511 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010257 6511 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010260 6511 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010264 6511 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010268 6511 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010271 6511 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010274 6511 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010278 6511 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010281 6511 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010284 6511 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010287 6511 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010290 6511 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010293 6511 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010296 6511 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010300 6511 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010306 6511 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:17:06.018255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010308 6511 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010311 6511 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010314 6511 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010318 6511 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010322 6511 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010325 6511 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010328 6511 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010331 6511 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010334 6511 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010337 6511 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010339 6511 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010342 6511 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010345 6511 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010350 6511 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010353 6511 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010356 6511 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010359 6511 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010362 6511 flags.go:64] FLAG: --enable-server="true" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010365 6511 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010369 6511 flags.go:64] FLAG: --event-burst="100" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010372 6511 flags.go:64] FLAG: --event-qps="50" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010375 6511 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010378 6511 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010381 6511 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:17:06.019293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010385 6511 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010388 6511 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010391 6511 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010394 6511 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010397 6511 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010400 6511 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010404 6511 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010407 6511 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010426 6511 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010430 6511 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010433 6511 flags.go:64] FLAG: --feature-gates="" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010437 6511 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010441 6511 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010444 6511 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010447 6511 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010450 6511 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010453 6511 flags.go:64] FLAG: --help="false" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010456 6511 flags.go:64] FLAG: --hostname-override="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010459 6511 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010462 6511 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010465 6511 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010468 6511 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010471 6511 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:17:06.020164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010474 6511 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010477 6511 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010480 6511 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010482 6511 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010485 6511 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010488 6511 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010491 6511 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010493 6511 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010496 6511 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010499 6511 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010502 6511 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010504 6511 flags.go:64] FLAG: --lock-file="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010507 6511 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010510 6511 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010513 6511 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010519 6511 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010522 6511 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010525 6511 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010528 6511 flags.go:64] FLAG: --logging-format="text" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010531 6511 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010534 6511 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010537 6511 flags.go:64] FLAG: --manifest-url="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010542 6511 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010546 6511 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010549 6511 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:17:06.021090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010552 6511 flags.go:64] FLAG: --max-pods="110" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010555 6511 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010558 6511 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010561 6511 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010564 6511 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010567 6511 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010571 6511 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010574 6511 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010581 6511 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010584 6511 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010587 6511 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010590 6511 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010593 6511 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010596 6511 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010599 6511 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010602 6511 flags.go:64] FLAG: --port="10250" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010604 6511 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010608 6511 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-06955bed59c5b342a" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010611 6511 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010614 6511 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010617 6511 flags.go:64] FLAG: --register-node="true" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010619 6511 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010627 6511 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010630 6511 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010633 6511 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010636 6511 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:17:06.022524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010639 6511 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010642 6511 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010645 6511 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010648 6511 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010651 6511 flags.go:64] FLAG: --runonce="false" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010654 6511 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010657 6511 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010660 6511 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010663 6511 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010666 6511 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010669 6511 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010672 6511 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010676 6511 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010679 6511 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010681 6511 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010684 6511 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010687 6511 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010690 6511 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010693 6511 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010695 6511 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010700 6511 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010703 6511 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010706 6511 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010710 6511 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010713 6511 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:17:06.023700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010716 6511 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010718 6511 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010721 6511 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010725 6511 flags.go:64] FLAG: --v="2" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010730 6511 flags.go:64] FLAG: --version="false" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010733 6511 flags.go:64] FLAG: --vmodule="" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010737 6511 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.010740 6511 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010868 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010871 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010874 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010876 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010879 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010882 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010884 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010887 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010889 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010892 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010894 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010899 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010901 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010904 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.024782 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010906 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010909 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010911 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010913 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010916 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010918 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010921 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010923 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010926 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010928 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010931 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010933 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010936 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010940 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010943 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010945 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010948 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010950 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010953 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010955 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.025476 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010958 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010961 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010963 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010966 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010969 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010971 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010974 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010976 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010978 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010982 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010985 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010987 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010990 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010992 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010995 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.010997 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011000 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011002 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011004 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011007 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.026005 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011009 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011012 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011014 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011017 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011019 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011023 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011025 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011028 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011030 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011033 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011035 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011037 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011040 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011042 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011045 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011048 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011051 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011053 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011056 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011058 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.026591 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011061 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011065 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011068 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011070 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011073 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011075 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011077 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011081 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011085 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011087 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011090 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011092 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011095 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011097 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011100 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011102 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011105 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011109 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011112 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.027081 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011115 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011118 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011121 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011123 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011126 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011128 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011131 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011133 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011136 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011139 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011142 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011144 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011147 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.011171 6511 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011205 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.027632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011210 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011213 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011215 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011218 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011221 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011223 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011227 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011230 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011233 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011236 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011239 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011242 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011245 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011247 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011250 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011254 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011256 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011259 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011261 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.028093 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011264 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011266 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011268 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011271 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011274 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011276 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011279 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011281 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011283 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011286 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011289 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011292 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011294 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011298 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011301 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011303 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011306 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011308 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011311 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011313 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.028648 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011316 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011318 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011320 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011323 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011325 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011328 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011330 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011333 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011337 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011339 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011342 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011344 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011347 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011349 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011351 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011354 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011357 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011359 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011362 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.029162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011364 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011367 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011369 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011372 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011374 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011377 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011380 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011383 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011386 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011388 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011391 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011394 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011396 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011398 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011401 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011403 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011406 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011408 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011427 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011430 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.029673 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011433 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011437 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011440 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011442 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011444 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011447 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011449 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011452 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011454 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011457 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011459 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011462 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011464 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011467 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011469 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011471 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011475 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011477 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011481 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.030165 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011484 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011486 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011488 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011491 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011496 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011499 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011502 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.011505 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.015911 6511 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.015931 6511 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.015956 6511 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.015968 6511 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016055 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016062 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016067 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016072 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016077 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016081 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016086 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.030701 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016090 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016094 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016099 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016103 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016107 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016111 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016115 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016119 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016123 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016128 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016132 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016136 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016140 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016145 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016149 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016153 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016158 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016163 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016167 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.031186 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016172 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016176 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016181 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016185 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016189 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016194 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016198 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016203 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016208 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016212 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016216 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016220 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016226 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016230 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016234 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016239 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016243 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016247 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016251 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016255 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.031663 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016260 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016264 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016268 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016272 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016276 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016280 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016284 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016288 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016292 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016296 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016300 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016306 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016311 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016315 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016319 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016324 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016328 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016332 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.032154 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016337 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016341 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016345 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016349 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016353 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016358 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016362 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016367 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016371 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016375 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016379 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016385 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016392 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016396 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016400 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016405 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016409 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016430 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016434 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016438 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.032599 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016442 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016446 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016449 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016453 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016457 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016462 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016467 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016471 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016476 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016480 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016484 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016489 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016495 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016501 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016506 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016510 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016515 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016520 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016524 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016528 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.033068 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016532 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016536 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.016580 6511 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016809 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016817 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016822 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016827 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016831 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016836 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016840 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016844 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016849 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016856 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016862 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016867 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.033633 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016871 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016876 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016880 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016885 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016890 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016894 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016899 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016903 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016907 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016911 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016915 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016919 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016923 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016927 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016932 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016936 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016940 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016944 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016948 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016953 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.034117 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016957 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016961 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016965 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016969 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016973 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016977 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016981 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016986 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016995 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.016999 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017004 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017008 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017012 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017017 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017021 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017025 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017029 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017034 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017038 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017043 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.034818 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017047 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017051 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017055 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017060 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017064 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017068 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017072 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017076 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017080 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017084 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017088 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017093 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017097 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017101 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017106 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017110 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017114 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017118 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017123 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017127 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.035517 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017133 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017137 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017142 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017146 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017151 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017155 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017159 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017163 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017167 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017171 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017177 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017181 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017185 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017189 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017194 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017198 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017202 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017206 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017211 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017215 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.036286 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017219 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017224 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017228 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017232 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017237 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017244 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017250 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017255 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017259 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017264 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017268 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017283 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017289 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017293 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.017337 6511 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:17:06.036897 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017503 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017512 6511 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017517 6511 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017521 6511 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017526 6511 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017530 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017534 6511 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017539 6511 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017544 6511 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017549 6511 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017553 6511 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017557 6511 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017561 6511 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017565 6511 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017570 6511 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017574 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017578 6511 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017583 6511 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017587 6511 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017591 6511 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:17:06.037260 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017602 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017607 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017611 6511 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017616 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017620 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017624 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017628 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017632 6511 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017637 6511 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017641 6511 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017647 6511 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017652 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017657 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017661 6511 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017665 6511 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017669 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017674 6511 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017678 6511 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017682 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:17:06.037794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017686 6511 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017690 6511 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017695 6511 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017700 6511 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017705 6511 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017709 6511 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017713 6511 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017717 6511 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017721 6511 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017725 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017729 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017733 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017737 6511 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017742 6511 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017746 6511 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017750 6511 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017755 6511 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017759 6511 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017763 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017767 6511 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:17:06.038266 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017771 6511 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017775 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017779 6511 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017785 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017791 6511 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017795 6511 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017800 6511 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017804 6511 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017808 6511 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017812 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017816 6511 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017821 6511 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017825 6511 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017829 6511 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017833 6511 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017839 6511 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017843 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017847 6511 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017852 6511 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017856 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:17:06.039042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017860 6511 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017865 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017869 6511 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017874 6511 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017878 6511 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017883 6511 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017887 6511 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017893 6511 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017899 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017903 6511 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017908 6511 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017912 6511 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017916 6511 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017920 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017925 6511 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017931 6511 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017935 6511 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017940 6511 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017944 6511 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:17:06.039865 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017948 6511 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.017952 6511 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.018047 6511 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.018051 6511 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.018056 6511 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.018060 6511 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.018069 6511 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:06.018075 6511 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.018160 6511 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.021490 6511 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.021564 6511 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.021854 6511 server.go:1014] "Starting client certificate rotation" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.021973 6511 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.022022 6511 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:54 +0000 UTC" deadline="2028-07-16 09:29:28.932822296 +0000 UTC" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.022075 6511 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14990h12m22.910749046s" Oct 30 19:17:06.040563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.022640 6511 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:17:06.040991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.024436 6511 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:17:06.040991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.025696 6511 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:17:06.040991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.027601 6511 log.go:25] "Validated CRI v1 image API" Oct 30 19:17:06.040991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.028359 6511 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:17:06.040991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.030272 6511 fs.go:135] Filesystem UUIDs: map[1cad5309-31dc-4079-88f9-ceb757267700:/dev/nvme0n1p3 37ba7ff5-7b29-4005-bf75-e14279c53fb4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 19:17:06.041151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.030286 6511 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d255784a66cba04a1260b7894c80be5f96d82d7e15857fe994cc010b04af5e9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d255784a66cba04a1260b7894c80be5f96d82d7e15857fe994cc010b04af5e9a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d73776132b56f4a6ae32955eb64f598aee379e4ab6599c05a133466ee34b3ed0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d73776132b56f4a6ae32955eb64f598aee379e4ab6599c05a133466ee34b3ed0/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9ea2100d1af2c9de15a75a20df278853baa10d6daf6966471c3081eb9322694/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9ea2100d1af2c9de15a75a20df278853baa10d6daf6966471c3081eb9322694/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0368674a6b971596f3c3c40b063b5be6d75bfd87b75ef464142f963843c486e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0368674a6b971596f3c3c40b063b5be6d75bfd87b75ef464142f963843c486e/userdata/shm major:0 minor:381 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc/volumes/kubernetes.io~projected/kube-api-access-zflc8:{mountpoint:/var/lib/kubelet/pods/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc/volumes/kubernetes.io~projected/kube-api-access-zflc8 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e6ea1e8-21c6-48fb-beb1-7246750874a4/volumes/kubernetes.io~projected/kube-api-access-7hnsw:{mountpoint:/var/lib/kubelet/pods/0e6ea1e8-21c6-48fb-beb1-7246750874a4/volumes/kubernetes.io~projected/kube-api-access-7hnsw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18670897-26cd-499f-900b-8d02e8ded5d6/volumes/kubernetes.io~projected/kube-api-access-lpxvl:{mountpoint:/var/lib/kubelet/pods/18670897-26cd-499f-900b-8d02e8ded5d6/volumes/kubernetes.io~projected/kube-api-access-lpxvl major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32f31184-b3c4-4a0e-8976-1d91195570e8/volumes/kubernetes.io~projected/kube-api-access-t2ww2:{mountpoint:/var/lib/kubelet/pods/32f31184-b3c4-4a0e-8976-1d91195570e8/volumes/kubernetes.io~projected/kube-api-access-t2ww2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b0c721a-72d7-44ce-af65-457dc171bfca/volumes/kubernetes.io~projected/kube-api-access-27jkz:{mountpoint:/var/lib/kubelet/pods/3b0c721a-72d7-44ce-af65-457dc171bfca/volumes/kubernetes.io~projected/kube-api-access-27jkz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b0c721a-72d7-44ce-af65-457dc171bfca/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3b0c721a-72d7-44ce-af65-457dc171bfca/volumes/kubernetes.io~secret/metrics-certs major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68413742-d011-4c65-8ce8-2cf80d43ac1a/volumes/kubernetes.io~projected/kube-api-access-k8wbv:{mountpoint:/var/lib/kubelet/pods/68413742-d011-4c65-8ce8-2cf80d43ac1a/volumes/kubernetes.io~projected/kube-api-access-k8wbv major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68413742-d011-4c65-8ce8-2cf80d43ac1a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/68413742-d011-4c65-8ce8-2cf80d43ac1a/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69df0eef-ee4f-4135-ab6c-01e806705d4a/volumes/kubernetes.io~projected/kube-api-access-z2ng9:{mountpoint:/var/lib/kubelet/pods/69df0eef-ee4f-4135-ab6c-01e806705d4a/volumes/kubernetes.io~projected/kube-api-access-z2ng9 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69df0eef-ee4f-4135-ab6c-01e806705d4a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/69df0eef-ee4f-4135-ab6c-01e806705d4a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~projected/kube-api-access-wr2l6:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~projected/kube-api-access-wr2l6 major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7/volumes/kubernetes.io~projected/kube-api-access-8fdwt:{mountpoint:/var/lib/kubelet/pods/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7/volumes/kubernetes.io~projected/kube-api-access-8fdwt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~projected/kube-api-access-9qxt7:{mountpoint:/var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~projected/kube-api-access-9qxt7 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/config-reloader/2 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/prometheus/15 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~projected/kube-api-access-xxc8v:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~projected/kube-api-access-xxc8v major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~projected/tls-assets major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/config major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/web-config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6/volumes/kubernetes.io~projected/kube-api-access-vkfb6:{mountpoint:/var/lib/kubelet/pods/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6/volumes/kubernetes.io~projected/kube-api-access-vkfb6 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volumes/kubernetes.io~projected/kube-api-access-m6wl9:{mountpoint:/var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volumes/kubernetes.io~projected/kube-api-access-m6wl9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~projected/kube-api-access-qs8s7:{mountpoint:/var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~projected/kube-api-access-qs8s7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0eda631-51b9-4c18-ad28-99ebb93e0db6/volumes/kubernetes.io~projected/kube-api-access-bsvcx:{mountpoint:/var/lib/kubelet/pods/d0eda631-51b9-4c18-ad28-99ebb93e0db6/volumes/kubernetes.io~projected/kube-api-access-bsvcx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0eda631-51b9-4c18-ad28-99ebb93e0db6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d0eda631-51b9-4c18-ad28-99ebb93e0db6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da8143dd-2ae4-4932-91b9-9215cf9cd469/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/da8143dd-2ae4-4932-91b9-9215cf9cd469/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e00ba2b0-311f-4ab1-9e26-e5020d81c211/volumes/kubernetes.io~projected/kube-api-access-2b22n:{mountpoint:/var/lib/kubelet/pods/e00ba2b0-311f-4ab1-9e26-e5020d81c211/volumes/kubernetes.io~projected/kube-api-access-2b22n major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~projected/kube-api-access-g5st4:{mountpoint:/var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~projected/kube-api-access-g5st4 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c7200ea057b08baff4240e4c642507d8e1fd8052924ebc27d0eaaadb0f4e2f41/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/486b2dd529703a1748ff7f03f0533f42885bbd9f9fea620fc068ef32eda8bb92/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/63a9b383f20dea2547bfaf3c788efa068e91642474fb854b7467f11cc37927d5/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9cc74824e5dd2cf08242d0f52fc67871c19f137a3787917cd7a6e37009db00bf/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3d43462f97bc001984d18e69a10dd805265e3fb845330eea64aa488c102dc278/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cdeca93a95466d061468d5b3014731206fb961e510f1f2a659e9ce98b0c8f5c2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a2ab0596997fe10fb1f7d4beee34d1fb2fa1e35ba9a692585ec5e97f6ac3910/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c1a1450ae9ad5ffc93e7e627819916663e0010b17963acbeba097b45b6846921/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2c2842f949758b357dc3cafb272fc9430605474bb3b4bdd40175f11ddfb73f16/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/36826b14a595b5aeb02ddf289e921b8d45a2efeb094d395197389fdfb1775e1f/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/399f0c165953dbe31057c135e0c369adfff881305aac35b76907b968ad08f272/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/e3234189dac7df31c25951be5a7dc9d69b0ab8277b0124e5acdb5af414ea58f5/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1c76a051ef12207695dd4537d430b809dd0c6caedfa70f3015947059efdcf938/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1e9c0a3727c8ede6555f86a1667105754e5b5cbe3f10a1e698c74b748fec62ac/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7f3af60f91cf5f9d2f72d26de453abfd785d093bee3665111f1108929bb51304/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/090988d836805b9f769177b6c69cdc69d2a971e85cba32f05bf7bb2c6ac5f1fa/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e3902778968d71209fa65d8f836678770f04f7bcb1fff7437c3e91dc72807c24/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9cb70475ffc8e9c06a3c1d834e73ca5899a2c300eeeb478b15fac3621ee168fa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ad77ede1eca3797f85ff896b17f6701cba7c242f02415af2c81aa05f08e7dda6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e6e2de1b40d2ada7a9f6da49ab368b12b612b0ebdfd33842df00d9810c9e98e0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/deb0e867e3d0e545c0cef3de3cf5f05a0fbc0e3d36486d0418f4cc8c848a8362/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/21e728ce2db97027867fdcb4ab963610c5ff340e2319a1e23c301f0a0a0eed60/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e9fd483052c3fc96e304c87a35350c4105697d82bd8faeadbe86e88064a74b81/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/705f21a999917a9b25da1445a1c0cafadc0f1304641dcb37ae0aba1f51fa77c3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1cc36995ac3e25e2cadd19d15f0b5427ff1a2fd093fe10ae6b196c4da3da79af/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/8a2812ad8a0c39c6ed07e7524bae636cc71223cdaaebc1bf9a27c69e428a91d1/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/bf8c4191d1e502a922318b3c974933351c648b0efe46babce7745cf97f324abb/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/b34d0be0c51b471c8552a83f227fe457c8875bb4ddee692d361899e22dc6f1a3/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d37f46ed4f902e8ff025ef3fcff3a7cdb49be7e398425dfd1b81dc538d65ebdd/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/38092daf3991729b0da58fb3e3b3c411e14dd54c05c6b66d4d25dc4be7181f10/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/98a9e480f6c4206f2e7fcc21379de7f3cc2096d3d46951039c2c4ef0f3a384ce/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/0367d0d2dd980273fc0246dc32e3a8755dd2e7bf477ab76200b94c2cde1a1656/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/55dba531165f74330bace4a260f43e51634895afff6b2e946446b36e65172de4/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/3c8ef9633dbc5e09db2b57b78514fbc2eb7e7c4170eb5ad69fb53becdc8c5668/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/bed129c3263d98820fb0da915067b124f9a6355dd484e88f2d30494d74f47bc5/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/9a9da3113122919c77d5412f0842292c76382aa0c535c8bd7171ef40be0af443/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/32fd129b08c2db1bcfe569a9b1c88d9f81a00a28f2beaf0108512abb8d40386a/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/94bea13147dd4c6ff8776bd011ea23567f2943a05b64d33c6633ad053fca218c/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/99389688df72e2756d4a2e8f11d038212f90c325dc7f98ed3f9ab45d681a9e6c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6c0bdeff6b2c780c4b57948d8aa8dc7ef94886a3bcb8da5a59c097e7ce608719/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/2aa7f72367918b4affe317988828e524b8340860bebe822eecd5ff07b5dd762e/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/3305dee0ed69aaad7a2dfd7758b882f9bcd4998afdacf1fb488830b3f073c574/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/66942fb07abf50085ad87a2e998a0b26b26ce9ec023a45f052ed7a8414e58495/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/52269e9a49ef6eecfacf227c7bec940b915c002f166f0f218ef506e13c02673e/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/413a38b37b7026edc23f3dd039e305b7d7a90037176c40e270a61ff65a14fcd7/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/03a60aa9c22beddcc92e9dd5f7cb738ed0f417446c684a90715a6b9cb01b4980/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/cec492200b10e6f8ffc9f51725eb5d6145c791efc3567f35fc9a983106104ab4/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/fe69fdea9de6b85f5087d22da5f2aa8799d85bca7bef6ea8342c0cac37692198/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/7a35a0f6a72bfd34bfb3ff82d684ea047a71843578855dc91af594b403bf73de/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/4903e8d8ea9c4297a92f9ef2982790150087336d85ed667f07713baeb9a117d3/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/27459e15c15033a09512da5550fa1bc6bd477d483ff233cd3a0703bd28cd085e/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/aca8185eeb74cfce9a67e4ed8c3949400a5abed7fadd2e24fcdc960359838079/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/f9e1bcc39cb534c1d512a7bc22fd45eb80314e27a30157bbca3319d239fec9f4/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/123f7656a5e1a3595c9cbf5be58a5276652f6fb7443330612c20c7309332a807/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/d7143735caa8b464ba0afa98b3d403ded19e868df9a3e792079487fd111df18a/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/f994fa4ae155fad6c1326b97cbf0b9d06457ec52e5f6cfe9a679d3a329ec7cff/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/248c3ae43792acd4affc3f860efe4b0d4a37a50ab95fb2edaf20a120893b61b0/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e829b6d44cbbcd93034ca9cc8c91f7d340df9f3400679fe6834fb4f7191aceec/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/379c4a2891343290d482ff1d1469fa183d534b4137e5933d1ff16b12c3d7782f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/58c48dc34b0386ee4adbfd51eeac8666297512f05af0431ea071689dd6bf27fb/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a7520a06633ee793692baf96bbbd3412237ef640c77466b8e44db5677c8cbb11/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/15fdbf8101bd246845d361fdd8e58d5ae187119b66f146c6e092b034ae09264b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/0e198688ea453aa7a10df8cd9a840a7c2c213de2d22e33f498f85442bc4f6a06/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c552bf0a745e73209df34ad01e487cd19d7771fc63765e6f2a497221bede7098/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/04a7ad26ee3db4b5da2a3a7574422fce604e25e7f5e30f51398f2fcc1fcfa9ff/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f810eea1e8b2002c638527b236346a21b69875d6258d5abe763c39f7696942fe/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/3a1c0302f8c88711beba8f9fbc288fa1cb66f32a365d40e0a18bd843ad35ee58/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037106 6511 manager.go:217] Machine: {Timestamp:2026-10-30 19:17:06.036657892 +0000 UTC m=+0.060261782 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113221 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe4d5f95420627167c9c68aaf947d SystemUUID:ec2fe4d5-f954-2062-7167-c9c68aaf947d BootID:ba493a55-7836-4585-8fb6-e1c037fbdb22 Filesystems:[{Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:335 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c676f79a516cc28108d4effa9bf9482c8471511e41ea300b52fdf098498f73cf/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/68413742-d011-4c65-8ce8-2cf80d43ac1a/volumes/kubernetes.io~projected/kube-api-access-k8wbv DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:335 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~projected/kube-api-access-qs8s7 DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:335 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:280 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/69df0eef-ee4f-4135-ab6c-01e806705d4a/volumes/kubernetes.io~projected/kube-api-access-z2ng9 DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9ea2100d1af2c9de15a75a20df278853baa10d6daf6966471c3081eb9322694/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/32f31184-b3c4-4a0e-8976-1d91195570e8/volumes/kubernetes.io~projected/kube-api-access-t2ww2 DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6/volumes/kubernetes.io~projected/kube-api-access-vkfb6 DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:337 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3b0c721a-72d7-44ce-af65-457dc171bfca/volumes/kubernetes.io~projected/kube-api-access-27jkz DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9943b13d1f7c750887d7850e93160b4e2d8c825bb22e0b08da4886a59b8a2945/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~projected/kube-api-access-wr2l6 DeviceMajor:0 DeviceMinor:281 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b0c721a-72d7-44ce-af65-457dc171bfca/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:375 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18670897-26cd-499f-900b-8d02e8ded5d6/volumes/kubernetes.io~projected/kube-api-access-lpxvl DeviceMajor:0 DeviceMinor:376 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab984142c74ab671d5172170d2fe33e6949b76b7f1626d9066fd20c9641f00e3/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:279 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0e6ea1e8-21c6-48fb-beb1-7246750874a4/volumes/kubernetes.io~projected/kube-api-access-7hnsw DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc/volumes/kubernetes.io~projected/kube-api-access-zflc8 DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d73776132b56f4a6ae32955eb64f598aee379e4ab6599c05a133466ee34b3ed0/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d255784a66cba04a1260b7894c80be5f96d82d7e15857fe994cc010b04af5e9a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:275 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:276 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19c1318166691eddab37c9cb36ff6861a2b572a2207968f642279642f6f0df56/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69ea1b29-9274-49ce-a4b1-81a8aa191296/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:278 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48c6b9f1ee6c292ea4ed6757b6011a318caea83c0983624c4292085142654d09/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:335 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0368674a6b971596f3c3c40b063b5be6d75bfd87b75ef464142f963843c486e/userdata/shm DeviceMajor:0 DeviceMinor:381 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68413742-d011-4c65-8ce8-2cf80d43ac1a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2adea89e12f200d26c1c1c545b926684ae10c781b66f8d670c9d90ed43700a18/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~projected/kube-api-access-9qxt7 DeviceMajor:0 DeviceMinor:230 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:330 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/69df0eef-ee4f-4135-ab6c-01e806705d4a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:329 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d87fcbfc4d9d64b7d94a5aefa638982079061bde06f030175014a47e8dc4425/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volumes/kubernetes.io~projected/kube-api-access-m6wl9 DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09646774a57fbf7034032e230c0f5d84b4f2fb624e765c2eb37d2a22e7537490/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ce0b21-0196-4d7c-9b36-99ec47e7d057/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:333 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~projected/kube-api-access-xxc8v DeviceMajor:0 DeviceMinor:338 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e00ba2b0-311f-4ab1-9e26-e5020d81c211/volumes/kubernetes.io~projected/kube-api-access-2b22n DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecfcc82c8ac18be6456d0053cc9e919848177cb88f7e8c722ccf1ba1cbbc2576/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:324 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/752b9e9a-aba1-4aee-8e54-83aa3ea6c854/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0eda631-51b9-4c18-ad28-99ebb93e0db6/volumes/kubernetes.io~projected/kube-api-access-bsvcx DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:332 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:336 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c741043edf8d670a9ccd992f4c602bdbd75922a5c7a578541bde1b6e11741860/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d24cb156c30dbebc45d53d24871bf7eb7a22f7695b899a9729efcdc703306b6/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~projected/kube-api-access-g5st4 DeviceMajor:0 DeviceMinor:270 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:334 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da8143dd-2ae4-4932-91b9-9215cf9cd469/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:371 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b70be47d-c307-4f47-b075-89a53633411d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:331 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ebdab24df6523ec088caf894a0037ad45c4c4f90250a22a852de8e33c94c725/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0eda631-51b9-4c18-ad28-99ebb93e0db6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7/volumes/kubernetes.io~projected/kube-api-access-8fdwt DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4585e9-d9f1-4736-aab7-485d22d6888c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1986d38c46e46483a1483d1655a4fc7106c9fe93b42512fbcf088d765ee3f643/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:19c1318166691ed MacAddress:ee:e9:c0:e5:29:8a Speed:10000 Mtu:8901} {Name:1d87fcbfc4d9d64 MacAddress:16:e1:50:da:d0:ad Speed:10000 Mtu:8901} {Name:2ce72a8106dec12 MacAddress:0a:f0:1a:30:cf:c1 Speed:10000 Mtu:8901} {Name:48c6b9f1ee6c292 MacAddress:de:36:9a:e6:02:19 Speed:10000 Mtu:8901} {Name:7d24cb156c30dbe MacAddress:de:8b:c1:6c:01:9f Speed:10000 Mtu:8901} {Name:9943b13d1f7c750 MacAddress:5e:83:0d:7e:eb:aa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cc:b8:a0:85 Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:65:d1:c4:98:62 Speed:0 Mtu:8901} {Name:d255784a66cba04 MacAddress:1e:cd:d4:0b:e2:47 Speed:10000 Mtu:8901} {Name:d9ea2100d1af2c9 MacAddress:9e:5d:cd:df:65:3b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cc:b8:a0:85 Speed:0 Mtu:9001} {Name:f0368674a6b9715 MacAddress:ea:80:ad:3c:7e:2d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:46:27:62:43:d8:df Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:6f:4a:b7:04:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037466 6511 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037507 6511 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037736 6511 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037755 6511 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-127.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037876 6511 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037884 6511 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037900 6511 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037912 6511 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.037970 6511 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038108 6511 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038208 6511 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038238 6511 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038251 6511 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038265 6511 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038274 6511 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.038282 6511 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.040747 6511 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.040792 6511 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:17:06.041507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.041495 6511 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:17:06.044030 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044016 6511 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:17:06.044030 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044029 6511 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:17:06.044137 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044126 6511 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:17:06.044341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044330 6511 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:17:06.044371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044361 6511 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:17:06.044487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044475 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:17:06.044524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044494 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:17:06.044524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044501 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:17:06.044524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044507 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:17:06.044524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044513 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:17:06.044524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044519 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:17:06.044524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044525 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044530 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044538 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044544 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044562 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044574 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044602 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:17:06.044753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.044608 6511 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:17:06.048016 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.048002 6511 server.go:1263] "Started kubelet" Oct 30 19:17:06.051141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.048095 6511 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:17:06.051141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.048401 6511 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:17:06.051141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.048518 6511 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:17:06.051141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.048754 6511 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:17:06.051141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.050213 6511 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:17:06.048570 ip-10-0-140-127 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054192 6511 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054367 6511 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054507 6511 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:56 +0000 UTC" deadline="2028-06-30 05:40:52.994157282 +0000 UTC" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054529 6511 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14602h23m46.939631024s" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054531 6511 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054546 6511 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:06.054545 6511 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-127.us-east-2.compute.internal\" not found" Oct 30 19:17:06.054761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.054546 6511 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:17:06.056256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056234 6511 factory.go:55] Registering systemd factory Oct 30 19:17:06.056256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056258 6511 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:17:06.056537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056524 6511 factory.go:153] Registering CRI-O factory Oct 30 19:17:06.056537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056537 6511 factory.go:223] Registration of the crio container factory successfully Oct 30 19:17:06.056631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056593 6511 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:17:06.056631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056612 6511 factory.go:103] Registering Raw factory Oct 30 19:17:06.056631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.056627 6511 manager.go:1199] Started watching for new ooms in manager Oct 30 19:17:06.057023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.057009 6511 manager.go:319] Starting recovery of all containers Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058246 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0eda631-51b9-4c18-ad28-99ebb93e0db6" volumeName="kubernetes.io/projected/d0eda631-51b9-4c18-ad28-99ebb93e0db6-kube-api-access-bsvcx" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058292 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e6ea1e8-21c6-48fb-beb1-7246750874a4" volumeName="kubernetes.io/projected/0e6ea1e8-21c6-48fb-beb1-7246750874a4-kube-api-access-7hnsw" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058311 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058328 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" volumeName="kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058348 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058363 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058385 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058402 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70be47d-c307-4f47-b075-89a53633411d" volumeName="kubernetes.io/projected/b70be47d-c307-4f47-b075-89a53633411d-kube-api-access-qs8s7" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058449 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32f31184-b3c4-4a0e-8976-1d91195570e8" volumeName="kubernetes.io/projected/32f31184-b3c4-4a0e-8976-1d91195570e8-kube-api-access-t2ww2" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058464 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b0c721a-72d7-44ce-af65-457dc171bfca" volumeName="kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058478 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058504 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4585e9-d9f1-4736-aab7-485d22d6888c" volumeName="kubernetes.io/projected/ff4585e9-d9f1-4736-aab7-485d22d6888c-kube-api-access-g5st4" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058521 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058543 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058583 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00a389a-2d47-4c51-9e90-85e9ffe3c9d6" volumeName="kubernetes.io/projected/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-kube-api-access-vkfb6" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058610 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc" volumeName="kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cni-binary-copy" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058626 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752b9e9a-aba1-4aee-8e54-83aa3ea6c854" volumeName="kubernetes.io/configmap/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058641 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752b9e9a-aba1-4aee-8e54-83aa3ea6c854" volumeName="kubernetes.io/projected/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-kube-api-access-9qxt7" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058661 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058675 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058695 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" volumeName="kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-env-overrides" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058710 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" volumeName="kubernetes.io/projected/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-kube-api-access-m6wl9" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058733 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da8143dd-2ae4-4932-91b9-9215cf9cd469" volumeName="kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058751 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e6ea1e8-21c6-48fb-beb1-7246750874a4" volumeName="kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058771 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4585e9-d9f1-4736-aab7-485d22d6888c" volumeName="kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058796 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4585e9-d9f1-4736-aab7-485d22d6888c" volumeName="kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-tls" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058818 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32f31184-b3c4-4a0e-8976-1d91195570e8" volumeName="kubernetes.io/empty-dir/32f31184-b3c4-4a0e-8976-1d91195570e8-tmp-dir" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058841 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out" seLinuxMountContext="" Oct 30 19:17:06.058882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058879 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058899 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0eda631-51b9-4c18-ad28-99ebb93e0db6" volumeName="kubernetes.io/secret/d0eda631-51b9-4c18-ad28-99ebb93e0db6-metrics-tls" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058916 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69df0eef-ee4f-4135-ab6c-01e806705d4a" volumeName="kubernetes.io/projected/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-api-access-z2ng9" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058936 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/configmap/69ea1b29-9274-49ce-a4b1-81a8aa191296-metrics-client-ca" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058954 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058967 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058986 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" volumeName="kubernetes.io/secret/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058999 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4585e9-d9f1-4736-aab7-485d22d6888c" volumeName="kubernetes.io/empty-dir/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-textfile" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059017 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059032 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752b9e9a-aba1-4aee-8e54-83aa3ea6c854" volumeName="kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-tls" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059048 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059070 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc" volumeName="kubernetes.io/projected/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-kube-api-access-zflc8" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059088 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e6ea1e8-21c6-48fb-beb1-7246750874a4" volumeName="kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059113 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e6ea1e8-21c6-48fb-beb1-7246750874a4" volumeName="kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-binary-copy" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059130 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68413742-d011-4c65-8ce8-2cf80d43ac1a" volumeName="kubernetes.io/projected/68413742-d011-4c65-8ce8-2cf80d43ac1a-kube-api-access-k8wbv" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059146 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69df0eef-ee4f-4135-ab6c-01e806705d4a" volumeName="kubernetes.io/configmap/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059172 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059191 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4" volumeName="kubernetes.io/secret/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-agent-certs" seLinuxMountContext="" Oct 30 19:17:06.059888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.058385 6511 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:06.061167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.059212 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70be47d-c307-4f47-b075-89a53633411d" volumeName="kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-tmp" seLinuxMountContext="" Oct 30 19:17:06.061203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061183 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:17:06.061244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061200 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0eda631-51b9-4c18-ad28-99ebb93e0db6" volumeName="kubernetes.io/configmap/d0eda631-51b9-4c18-ad28-99ebb93e0db6-config-volume" seLinuxMountContext="" Oct 30 19:17:06.061244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061214 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4585e9-d9f1-4736-aab7-485d22d6888c" volumeName="kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 19:17:06.061244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061227 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/projected/69ea1b29-9274-49ce-a4b1-81a8aa191296-kube-api-access-wr2l6" seLinuxMountContext="" Oct 30 19:17:06.061244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061242 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7" volumeName="kubernetes.io/projected/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-kube-api-access-8fdwt" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061251 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061260 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061277 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061288 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a00a389a-2d47-4c51-9e90-85e9ffe3c9d6" volumeName="kubernetes.io/configmap/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-serviceca" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061349 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b0c721a-72d7-44ce-af65-457dc171bfca" volumeName="kubernetes.io/projected/3b0c721a-72d7-44ce-af65-457dc171bfca-kube-api-access-27jkz" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061365 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69df0eef-ee4f-4135-ab6c-01e806705d4a" volumeName="kubernetes.io/secret/69df0eef-ee4f-4135-ab6c-01e806705d4a-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 19:17:06.061401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061379 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7" volumeName="kubernetes.io/configmap/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061425 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061441 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061455 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4585e9-d9f1-4736-aab7-485d22d6888c" volumeName="kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061467 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/empty-dir/69ea1b29-9274-49ce-a4b1-81a8aa191296-tmp" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061481 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061495 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752b9e9a-aba1-4aee-8e54-83aa3ea6c854" volumeName="kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061507 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061524 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68413742-d011-4c65-8ce8-2cf80d43ac1a" volumeName="kubernetes.io/secret/68413742-d011-4c65-8ce8-2cf80d43ac1a-cert" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061538 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc" volumeName="kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-daemon-config" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061553 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18670897-26cd-499f-900b-8d02e8ded5d6" volumeName="kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061565 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69df0eef-ee4f-4135-ab6c-01e806705d4a" volumeName="kubernetes.io/empty-dir/69df0eef-ee4f-4135-ab6c-01e806705d4a-data-volume" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061579 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ea1b29-9274-49ce-a4b1-81a8aa191296" volumeName="kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061593 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4" volumeName="kubernetes.io/configmap/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-konnectivity-ca" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061611 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b70be47d-c307-4f47-b075-89a53633411d" volumeName="kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-etc-tuned" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061622 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e00ba2b0-311f-4ab1-9e26-e5020d81c211" volumeName="kubernetes.io/projected/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kube-api-access-2b22n" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061636 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7951e5f1-5cb5-4185-836d-599ac3d2223f" volumeName="kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061650 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" volumeName="kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-config" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061662 6511 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0eda631-51b9-4c18-ad28-99ebb93e0db6" volumeName="kubernetes.io/empty-dir/d0eda631-51b9-4c18-ad28-99ebb93e0db6-tmp-dir" seLinuxMountContext="" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061676 6511 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:17:06.061843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061687 6511 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:17:06.062832 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.061864 6511 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:06.062832 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.062409 6511 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:06.065805 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.065771 6511 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:17:06.083337 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.083315 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:17:06.083439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.083330 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:17:06.087505 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.087476 6511 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:17:06.089102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.089082 6511 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:17:06.089218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.089109 6511 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:17:06.089218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.089136 6511 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:17:06.089218 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:06.089177 6511 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:17:06.093764 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.093743 6511 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:17:06.097211 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.097183 6511 generic.go:356] "Generic (PLEG): container finished" podID="ff4585e9-d9f1-4736-aab7-485d22d6888c" containerID="e1b34dbed9172ecddcd6374c5f58a8c3f81057b0618cfcaf22842d6e672e8d38" exitCode=0 Oct 30 19:17:06.115918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.115892 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="22f8592a5f4c026ae3e51cca4ca9f66fdee9dc8a25b9109ef558133f58c3d0ca" exitCode=0 Oct 30 19:17:06.116024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.115919 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="811fc10ec793a03cab860afcdefef4a5d0a2c524f330ced4a9177a0759775729" exitCode=0 Oct 30 19:17:06.116024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.115930 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="fd9ca76bf230cd3772e229f86df4a5c2a178703eafe4dff1c5572ea986073cbc" exitCode=0 Oct 30 19:17:06.116024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.115941 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="c60313d7d4b38cb9cac62bef3f436f481e9a8b7525ee7ae26c519c3c80e932f9" exitCode=0 Oct 30 19:17:06.116024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.115951 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="915a9be68ffa921c20eba306e56879cc5753c141012fe92d4178558fc33afef7" exitCode=0 Oct 30 19:17:06.116024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.115961 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e6ea1e8-21c6-48fb-beb1-7246750874a4" containerID="ec84063e2c92ac255829233e073c913cdfbe96bd2783e40772be160730fd2a42" exitCode=0 Oct 30 19:17:06.117933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.117912 6511 generic.go:356] "Generic (PLEG): container finished" podID="62885df979c25d07a503fc33848fc4ff" containerID="fdbcf6a69de0734f0724543db584efde6bfcea114b42e7b7af3a8890597d9d29" exitCode=0 Oct 30 19:17:06.124745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.124717 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" exitCode=0 Oct 30 19:17:06.130054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.130037 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:17:06.130643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.130614 6511 generic.go:356] "Generic (PLEG): container finished" podID="a3ce0b21-0196-4d7c-9b36-99ec47e7d057" containerID="fadeb12c6aa6b5ae9923a81221dbee4bbb308dac755b181d2393c0e5d027faef" exitCode=1 Oct 30 19:17:06.156643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.156626 6511 manager.go:324] Recovery completed Oct 30 19:17:06.177712 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177648 6511 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:17:06.177712 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177666 6511 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:17:06.177712 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177685 6511 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:17:06.177918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177834 6511 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:17:06.177918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177849 6511 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:17:06.177918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177890 6511 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:17:06.177918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177899 6511 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:17:06.177918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177908 6511 policy_none.go:50] "Start" Oct 30 19:17:06.178143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177928 6511 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:17:06.178143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.177941 6511 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:17:06.178143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.178119 6511 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:17:06.178143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.178130 6511 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:17:06.178143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.178138 6511 policy_none.go:44] "Start" Oct 30 19:17:06.182661 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.182637 6511 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:17:06.185258 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.185245 6511 manager.go:341] "Starting Device Plugin manager" Oct 30 19:17:06.185727 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:06.185269 6511 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:17:06.185727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.185728 6511 server.go:87] "Starting device plugin registration server" Oct 30 19:17:06.185981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.185971 6511 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:17:06.186019 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.185983 6511 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:17:06.186104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.186081 6511 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:17:06.186240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.186202 6511 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:17:06.186308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.186249 6511 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:17:06.186308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.186256 6511 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:17:06.186392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.186310 6511 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:17:06.186237551Z","UUID":"aa05ea44-a9cf-4a32-8ccd-4c545dc87933","Handler":null,"Name":"","Endpoint":""} Oct 30 19:17:06.188153 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:06.188125 6511 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:17:06.189355 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.189292 6511 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal"] Oct 30 19:17:06.189473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.189430 6511 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:17:06.189473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.189468 6511 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:17:06.190710 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.190524 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"62885df979c25d07a503fc33848fc4ff","Type":"ContainerStarted","Data":"c02648b332dd266084e43fe91bb4f63f0e6dfaab17a2faf1e5c1ea9ff09286a8"} Oct 30 19:17:06.190710 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.190586 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"62885df979c25d07a503fc33848fc4ff","Type":"ContainerDied","Data":"fdbcf6a69de0734f0724543db584efde6bfcea114b42e7b7af3a8890597d9d29"} Oct 30 19:17:06.190710 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.190607 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"62885df979c25d07a503fc33848fc4ff","Type":"ContainerStarted","Data":"d1d2bb3d034c2bc49c31517cf865028511c2691e8253ab59c4ba2493e3f7ed94"} Oct 30 19:17:06.190710 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.190653 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"4860c9cfd0c091391ec52fd8f7245c90","Type":"ContainerStarted","Data":"ca8785ade9db83bdcb904d9b2040b82211c3fb67e2f3b86802e011301dff3139"} Oct 30 19:17:06.190710 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.190672 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" event={"ID":"4860c9cfd0c091391ec52fd8f7245c90","Type":"ContainerStarted","Data":"856160f70f26a1fa81d7d897ba8301ab726d2836c6e328bf96c57319a614cc5c"} Oct 30 19:17:06.191166 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.190884 6511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.191549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.191531 6511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.205991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.205972 6511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:06.206071 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:06.206027 6511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.206261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.206249 6511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:06.206305 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:06.206274 6511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.266597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.266569 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.266707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.266599 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4860c9cfd0c091391ec52fd8f7245c90-config\") pod \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"4860c9cfd0c091391ec52fd8f7245c90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.266707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.266617 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.286679 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.286659 6511 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:17:06.288802 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.288784 6511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:17:06.288899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.288821 6511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:17:06.288899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.288837 6511 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-127.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:17:06.288899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.288891 6511 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.300709 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.300692 6511 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.300775 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.300749 6511 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.367767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.367737 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.367767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.367751 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.367943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.367784 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.367943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.367810 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4860c9cfd0c091391ec52fd8f7245c90-config\") pod \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"4860c9cfd0c091391ec52fd8f7245c90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.367943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.367853 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4860c9cfd0c091391ec52fd8f7245c90-config\") pod \"kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"4860c9cfd0c091391ec52fd8f7245c90\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:06.367943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:06.367886 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62885df979c25d07a503fc33848fc4ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" (UID: \"62885df979c25d07a503fc33848fc4ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:07.038425 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.038376 6511 apiserver.go:52] "Watching apiserver" Oct 30 19:17:07.054868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.054838 6511 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:17:07.055882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.055846 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-gh6sj","kube-system/kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-468t5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th","openshift-ovn-kubernetes/ovnkube-node-nbd9r","openshift-cluster-node-tuning-operator/tuned-8r4wz","openshift-monitoring/thanos-querier-7646648d5d-jd8l2","openshift-network-operator/iptables-alerter-vf6wx","kube-system/global-pull-secret-syncer-f6wtd","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/node-exporter-pfp6b","openshift-monitoring/prometheus-operator-9cd89b74c-7mc52","openshift-multus/multus-52x9m","openshift-dns/dns-default-5vdt7","openshift-ingress-canary/ingress-canary-4wmdq","openshift-image-registry/node-ca-4wfl7","openshift-insights/insights-runtime-extractor-kkdfr","kube-system/konnectivity-agent-hcv7r","openshift-dns/node-resolver-kxt77","openshift-network-diagnostics/network-check-target-9hzhx"] Oct 30 19:17:07.066474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066451 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:17:07.066572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066493 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.066572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066548 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.066684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066584 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.066760 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066743 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.066822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066776 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.066822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066786 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:17:07.066822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066780 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:17:07.066950 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.066831 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.067068 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.067052 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:17:07.067068 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.067062 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.068079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068055 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.068079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068077 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.068192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068077 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:17:07.068192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068085 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:17:07.068192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068088 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.068688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068673 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.068750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068711 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.068750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068722 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.068750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068744 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:17:07.068898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068744 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:17:07.068898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068761 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.068898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068784 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:17:07.068898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.068835 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.139941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.139914 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:17:07.140353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.140329 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:17:07.140490 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.140395 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:17:07.140675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.140636 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-6bckm\"" Oct 30 19:17:07.142320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.142300 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wctlt\"" Oct 30 19:17:07.142455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.142437 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:17:07.142521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.142479 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l9rvz\"" Oct 30 19:17:07.142587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.142517 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:17:07.143194 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.143181 6511 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:07.143504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.143488 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:17:07.143504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.143495 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:17:07.143504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.143491 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:17:07.143645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.143542 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:17:07.144009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.143994 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:17:07.144310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.144294 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:17:07.144367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.144330 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:17:07.144367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.144345 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:17:07.144367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.144360 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:17:07.144367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.144360 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:17:07.145222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.145199 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzjv5\"" Oct 30 19:17:07.145343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.145330 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bfjrl\"" Oct 30 19:17:07.145343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.145337 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2qhvn\"" Oct 30 19:17:07.145458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.145387 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-n7s42\"" Oct 30 19:17:07.145954 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.145941 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:17:07.146134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146122 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrdg\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146136 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146147 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lfmhh\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146163 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146162 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bc6qp\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146176 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2k8z2\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146199 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146179 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7b7kv\"" Oct 30 19:17:07.146208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146209 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:17:07.146454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.146216 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9tgzd\"" Oct 30 19:17:07.147215 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147197 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:17:07.147215 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147208 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rklsm\"" Oct 30 19:17:07.147374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147222 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-29o98lk1qr308\"" Oct 30 19:17:07.147374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147234 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rd2gbba2224\"" Oct 30 19:17:07.147374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147236 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2lpdx\"" Oct 30 19:17:07.147374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147314 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:17:07.147374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147337 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:17:07.147374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147345 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xph5z\"" Oct 30 19:17:07.147598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147347 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:17:07.147598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147398 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mglp5\"" Oct 30 19:17:07.147598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147406 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6xn8\"" Oct 30 19:17:07.147598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147438 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:17:07.147598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147447 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:17:07.147598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.147460 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qmpzj\"" Oct 30 19:17:07.153800 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.153783 6511 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:17:07.153875 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:07.153819 6511 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-127.us-east-2.compute.internal" Oct 30 19:17:07.156409 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.156390 6511 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:17:07.166784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.166762 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:17:07.166868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.166771 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.166868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.166819 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:17:07.166868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.166827 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.166868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.166860 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.167038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.166960 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:17:07.167038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.167030 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:17:07.167121 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.167067 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:17:07.167251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.167237 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.167251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.167248 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:17:07.167336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.167242 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.168143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.168128 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:17:07.168195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.168143 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:17:07.168195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.168182 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:17:07.168195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.168187 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:17:07.169166 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.169154 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:17:07.170557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170533 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:07.170656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170559 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.170656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170580 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-socket-dir-parent\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.170656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170595 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-etc-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.170656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170616 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.170853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170659 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.170853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170680 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jkz\" (UniqueName: \"kubernetes.io/projected/3b0c721a-72d7-44ce-af65-457dc171bfca-kube-api-access-27jkz\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:07.170853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170742 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-k8s-cni-cncf-io\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.170853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170826 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-konnectivity-ca\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:17:07.171027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170884 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32f31184-b3c4-4a0e-8976-1d91195570e8-hosts-file\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.171027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170925 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.171027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170940 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-netns\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.171027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.170936 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.171147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171040 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-iptables-alerter-script\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.171147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171070 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-metrics-client-ca\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.171147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171097 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.171147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171128 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.171147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171141 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-konnectivity-ca\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171155 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-kubernetes\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171182 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171239 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171287 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-metrics-client-ca\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171299 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-grpc-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171287 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-iptables-alerter-script\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.171368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171362 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-slash\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171372 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171387 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-netd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171402 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171447 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-textfile\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171483 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171505 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-textfile\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171492 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171559 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea1b29-9274-49ce-a4b1-81a8aa191296-tmp\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171613 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvcx\" (UniqueName: \"kubernetes.io/projected/d0eda631-51b9-4c18-ad28-99ebb93e0db6-kube-api-access-bsvcx\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171678 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea1b29-9274-49ce-a4b1-81a8aa191296-tmp\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.171726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171693 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zflc8\" (UniqueName: \"kubernetes.io/projected/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-kube-api-access-zflc8\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171739 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-node-log\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171763 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-grpc-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171786 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171815 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da8143dd-2ae4-4932-91b9-9215cf9cd469-original-pull-secret\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171819 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-sys\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171859 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171878 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ng9\" (UniqueName: \"kubernetes.io/projected/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-api-access-z2ng9\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171892 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171908 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.171924 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2l6\" (UniqueName: \"kubernetes.io/projected/69ea1b29-9274-49ce-a4b1-81a8aa191296-kube-api-access-wr2l6\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172025 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2ww2\" (UniqueName: \"kubernetes.io/projected/32f31184-b3c4-4a0e-8976-1d91195570e8-kube-api-access-t2ww2\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172058 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-kubelet\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172098 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-tmp\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172109 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.172174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172123 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-kubelet\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172192 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172201 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-tmp\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172233 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6wl9\" (UniqueName: \"kubernetes.io/projected/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-kube-api-access-m6wl9\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172251 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172266 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172280 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172300 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-script-lib\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172314 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172434 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172473 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172525 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-script-lib\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.172773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172530 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172680 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172519 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-host\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172916 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5st4\" (UniqueName: \"kubernetes.io/projected/ff4585e9-d9f1-4736-aab7-485d22d6888c-kube-api-access-g5st4\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172944 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172954 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-dbus\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172951 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.172982 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-serviceca\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173014 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69df0eef-ee4f-4135-ab6c-01e806705d4a-crio-socket\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173021 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173051 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-kubelet-config\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.173279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173247 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-tls\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.173778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173401 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-systemd\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.173778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173450 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-sys\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.173778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173506 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cnibin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.173778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173556 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-etc-kubernetes\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.173778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173609 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.173778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173691 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysconfig\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.174096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173823 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-bin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.174096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173862 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69ea1b29-9274-49ce-a4b1-81a8aa191296-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.174096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173892 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-multus-certs\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.174096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173909 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-serviceca\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.174096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.173946 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.174317 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174172 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.174317 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174223 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.174405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174350 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.174405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174386 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-systemd-units\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.174516 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174468 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.174516 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174496 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.174598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174533 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-hostroot\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.174598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174567 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-env-overrides\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.174767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174748 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-env-overrides\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.174862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174791 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.174862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174825 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.174862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174827 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.174862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174858 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-ovn\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174889 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.174976 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-modprobe-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175006 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-netns\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175044 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-config\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175078 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175093 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.175117 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175111 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-multus\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175137 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175235 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175250 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175287 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-systemd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175443 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175472 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175505 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175544 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-root\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.175613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175592 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8wbv\" (UniqueName: \"kubernetes.io/projected/68413742-d011-4c65-8ce8-2cf80d43ac1a-kube-api-access-k8wbv\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:17:07.176227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175628 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.176227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175638 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovnkube-config\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.176227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175662 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cnibin\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.176227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.175692 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-etc-tuned\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.177443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177400 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-tls\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.177521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177481 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.177580 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177552 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0eda631-51b9-4c18-ad28-99ebb93e0db6-metrics-tls\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.177626 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177602 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.177674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177630 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-bin\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.177731 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177709 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b70be47d-c307-4f47-b075-89a53633411d-etc-tuned\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.177794 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177763 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.177827 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177772 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-tls\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.177864 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177844 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0eda631-51b9-4c18-ad28-99ebb93e0db6-metrics-tls\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.177895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177861 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-system-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.177935 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177915 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-socket-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.177970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177920 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.177970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177936 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-etc-selinux\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.178047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.177972 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-host\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.178047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178007 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0eda631-51b9-4c18-ad28-99ebb93e0db6-tmp-dir\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.178154 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178143 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0eda631-51b9-4c18-ad28-99ebb93e0db6-tmp-dir\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.178203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178141 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.178203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178182 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-run\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.178300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178226 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-var-lib-kubelet\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.178300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178246 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.178643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178269 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-lib-modules\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.178703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178673 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cni-binary-copy\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.178703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178677 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.178785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178703 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-daemon-config\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.178785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178740 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.178785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178771 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-wtmp\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.178785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178774 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178798 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178837 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178867 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovn-node-metrics-cert\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178880 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178893 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ea1b29-9274-49ce-a4b1-81a8aa191296-metrics-client-ca\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178918 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178924 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-daemon-config\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178923 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cni-binary-copy\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178947 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b22n\" (UniqueName: \"kubernetes.io/projected/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kube-api-access-2b22n\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.178980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.178974 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxt7\" (UniqueName: \"kubernetes.io/projected/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-kube-api-access-9qxt7\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179001 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179026 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f31184-b3c4-4a0e-8976-1d91195570e8-tmp-dir\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179051 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8s7\" (UniqueName: \"kubernetes.io/projected/b70be47d-c307-4f47-b075-89a53633411d-kube-api-access-qs8s7\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179075 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-var-lib-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179083 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69ea1b29-9274-49ce-a4b1-81a8aa191296-metrics-client-ca\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179060 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179099 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fdwt\" (UniqueName: \"kubernetes.io/projected/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-kube-api-access-8fdwt\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179114 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179124 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179149 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179168 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-ovn-node-metrics-cert\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179173 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-registration-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179203 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-sys-fs\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179206 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179230 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-host-slash\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179254 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-conf-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179280 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179304 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179476 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-conf\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179510 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68413742-d011-4c65-8ce8-2cf80d43ac1a-cert\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179536 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69df0eef-ee4f-4135-ab6c-01e806705d4a-data-volume\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179563 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eda631-51b9-4c18-ad28-99ebb93e0db6-config-volume\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179590 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-device-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179618 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69df0eef-ee4f-4135-ab6c-01e806705d4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179648 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfb6\" (UniqueName: \"kubernetes.io/projected/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-kube-api-access-vkfb6\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179679 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-system-cni-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179717 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-log-socket\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179737 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2ww2\" (UniqueName: \"kubernetes.io/projected/32f31184-b3c4-4a0e-8976-1d91195570e8-kube-api-access-t2ww2\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179746 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-agent-certs\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179779 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179789 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnsw\" (UniqueName: \"kubernetes.io/projected/0e6ea1e8-21c6-48fb-beb1-7246750874a4-kube-api-access-7hnsw\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.179719 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvcx\" (UniqueName: \"kubernetes.io/projected/d0eda631-51b9-4c18-ad28-99ebb93e0db6-kube-api-access-bsvcx\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180000 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6wl9\" (UniqueName: \"kubernetes.io/projected/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-kube-api-access-m6wl9\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180033 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxvl\" (UniqueName: \"kubernetes.io/projected/18670897-26cd-499f-900b-8d02e8ded5d6-kube-api-access-lpxvl\") pod \"network-check-target-9hzhx\" (UID: \"18670897-26cd-499f-900b-8d02e8ded5d6\") " pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180070 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4-agent-certs\") pod \"konnectivity-agent-hcv7r\" (UID: \"950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4\") " pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180001 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f31184-b3c4-4a0e-8976-1d91195570e8-tmp-dir\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180121 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69df0eef-ee4f-4135-ab6c-01e806705d4a-data-volume\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180193 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5st4\" (UniqueName: \"kubernetes.io/projected/ff4585e9-d9f1-4736-aab7-485d22d6888c-kube-api-access-g5st4\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180235 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b0c721a-72d7-44ce-af65-457dc171bfca-metrics-certs\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180278 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zflc8\" (UniqueName: \"kubernetes.io/projected/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-kube-api-access-zflc8\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180286 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0eda631-51b9-4c18-ad28-99ebb93e0db6-config-volume\") pod \"dns-default-5vdt7\" (UID: \"d0eda631-51b9-4c18-ad28-99ebb93e0db6\") " pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180309 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69df0eef-ee4f-4135-ab6c-01e806705d4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180471 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180458 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68413742-d011-4c65-8ce8-2cf80d43ac1a-cert\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:17:07.180638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180550 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.181381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180690 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ng9\" (UniqueName: \"kubernetes.io/projected/69df0eef-ee4f-4135-ab6c-01e806705d4a-kube-api-access-z2ng9\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.181381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180783 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2l6\" (UniqueName: \"kubernetes.io/projected/69ea1b29-9274-49ce-a4b1-81a8aa191296-kube-api-access-wr2l6\") pod \"thanos-querier-7646648d5d-jd8l2\" (UID: \"69ea1b29-9274-49ce-a4b1-81a8aa191296\") " pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:07.181381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.180835 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jkz\" (UniqueName: \"kubernetes.io/projected/3b0c721a-72d7-44ce-af65-457dc171bfca-kube-api-access-27jkz\") pod \"network-metrics-daemon-468t5\" (UID: \"3b0c721a-72d7-44ce-af65-457dc171bfca\") " pod="openshift-multus/network-metrics-daemon-468t5" Oct 30 19:17:07.182492 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.182475 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.185115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.185095 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.185393 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.185375 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8wbv\" (UniqueName: \"kubernetes.io/projected/68413742-d011-4c65-8ce8-2cf80d43ac1a-kube-api-access-k8wbv\") pod \"ingress-canary-4wmdq\" (UID: \"68413742-d011-4c65-8ce8-2cf80d43ac1a\") " pod="openshift-ingress-canary/ingress-canary-4wmdq" Oct 30 19:17:07.192923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.192904 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8s7\" (UniqueName: \"kubernetes.io/projected/b70be47d-c307-4f47-b075-89a53633411d-kube-api-access-qs8s7\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.193264 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.193246 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") pod \"prometheus-k8s-0\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:07.193692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.193674 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfb6\" (UniqueName: \"kubernetes.io/projected/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-kube-api-access-vkfb6\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.193751 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.193725 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fdwt\" (UniqueName: \"kubernetes.io/projected/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-kube-api-access-8fdwt\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.194575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.194559 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxt7\" (UniqueName: \"kubernetes.io/projected/752b9e9a-aba1-4aee-8e54-83aa3ea6c854-kube-api-access-9qxt7\") pod \"prometheus-operator-9cd89b74c-7mc52\" (UID: \"752b9e9a-aba1-4aee-8e54-83aa3ea6c854\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7mc52" Oct 30 19:17:07.194762 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.194748 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b22n\" (UniqueName: \"kubernetes.io/projected/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kube-api-access-2b22n\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.202521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.202504 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnsw\" (UniqueName: \"kubernetes.io/projected/0e6ea1e8-21c6-48fb-beb1-7246750874a4-kube-api-access-7hnsw\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.281222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281192 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-socket-dir-parent\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281230 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-etc-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281255 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281278 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-k8s-cni-cncf-io\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281300 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32f31184-b3c4-4a0e-8976-1d91195570e8-hosts-file\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281301 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-socket-dir-parent\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281353 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-k8s-cni-cncf-io\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281356 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281339 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-etc-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281386 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-netns\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281430 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-kubernetes\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281467 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32f31184-b3c4-4a0e-8976-1d91195570e8-hosts-file\") pod \"node-resolver-kxt77\" (UID: \"32f31184-b3c4-4a0e-8976-1d91195570e8\") " pod="openshift-dns/node-resolver-kxt77" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281471 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-slash\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281503 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-slash\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281471 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-netns\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281524 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-netd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281523 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-kubernetes\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281558 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-netd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281564 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-node-log\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281597 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-node-log\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281612 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281643 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-sys\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281686 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281686 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281710 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-sys\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281713 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-kubelet\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281733 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-kubelet\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281742 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-kubelet\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281740 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281769 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-host\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281776 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-kubelet\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281786 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-dbus\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281802 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69df0eef-ee4f-4135-ab6c-01e806705d4a-crio-socket\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281808 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-host\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281824 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-kubelet-config\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281840 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-systemd\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281848 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-dbus\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281873 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-sys\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281881 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-systemd\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281887 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da8143dd-2ae4-4932-91b9-9215cf9cd469-kubelet-config\") pod \"global-pull-secret-syncer-f6wtd\" (UID: \"da8143dd-2ae4-4932-91b9-9215cf9cd469\") " pod="kube-system/global-pull-secret-syncer-f6wtd" Oct 30 19:17:07.281878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281874 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69df0eef-ee4f-4135-ab6c-01e806705d4a-crio-socket\") pod \"insights-runtime-extractor-kkdfr\" (UID: \"69df0eef-ee4f-4135-ab6c-01e806705d4a\") " pod="openshift-insights/insights-runtime-extractor-kkdfr" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281896 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cnibin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281913 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-sys\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281925 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-cnibin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281925 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-etc-kubernetes\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281948 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysconfig\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281946 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-etc-kubernetes\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.281964 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-bin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282001 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-multus-certs\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282004 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysconfig\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282028 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-systemd-units\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282028 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-bin\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282045 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-hostroot\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282065 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-ovn\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282084 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-modprobe-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282103 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-netns\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282104 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-run-multus-certs\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282117 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-hostroot\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282141 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-systemd-units\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282149 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-multus\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282130 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-ovn\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282124 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-host-var-lib-cni-multus\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282159 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-netns\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282187 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-systemd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282204 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282224 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-modprobe-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282229 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-root\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282244 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-systemd\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282248 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-root\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282263 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cnibin\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282288 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-bin\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282303 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282318 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-system-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282264 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-run-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282341 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-socket-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282361 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-cnibin\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282365 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-etc-selinux\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282395 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-host\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282437 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-etc-selinux\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282444 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-run\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282468 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-var-lib-kubelet\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282481 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-cni-bin\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282492 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-lib-modules\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282519 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-wtmp\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282540 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282562 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282589 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-d\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282623 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a00a389a-2d47-4c51-9e90-85e9ffe3c9d6-host\") pod \"node-ca-4wfl7\" (UID: \"a00a389a-2d47-4c51-9e90-85e9ffe3c9d6\") " pod="openshift-image-registry/node-ca-4wfl7" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282624 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-var-lib-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282670 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-system-cni-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282712 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-var-lib-kubelet\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282742 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-lib-modules\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282777 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-socket-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282588 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-var-lib-openvswitch\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282789 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff4585e9-d9f1-4736-aab7-485d22d6888c-node-exporter-wtmp\") pod \"node-exporter-pfp6b\" (UID: \"ff4585e9-d9f1-4736-aab7-485d22d6888c\") " pod="openshift-monitoring/node-exporter-pfp6b" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282673 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-run\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282835 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-host-run-ovn-kubernetes\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.282821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282840 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282862 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-registration-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282899 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-sys-fs\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282924 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-host-slash\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282946 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-conf-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282955 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-registration-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282969 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-conf\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282995 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-device-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283021 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-system-cni-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283031 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-sys-fs\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283044 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-log-socket\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283074 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e00ba2b0-311f-4ab1-9e26-e5020d81c211-device-dir\") pod \"aws-ebs-csi-driver-node-bg8th\" (UID: \"e00ba2b0-311f-4ab1-9e26-e5020d81c211\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bg8th" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.282996 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7-host-slash\") pod \"iptables-alerter-vf6wx\" (UID: \"6a35f2ba-0f4b-41e8-b3b5-6ba6ff4d62a7\") " pod="openshift-network-operator/iptables-alerter-vf6wx" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283108 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3ce0b21-0196-4d7c-9b36-99ec47e7d057-log-socket\") pod \"ovnkube-node-nbd9r\" (UID: \"a3ce0b21-0196-4d7c-9b36-99ec47e7d057\") " pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283112 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e6ea1e8-21c6-48fb-beb1-7246750874a4-system-cni-dir\") pod \"multus-additional-cni-plugins-gh6sj\" (UID: \"0e6ea1e8-21c6-48fb-beb1-7246750874a4\") " pod="openshift-multus/multus-additional-cni-plugins-gh6sj" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283147 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc-multus-conf-dir\") pod \"multus-52x9m\" (UID: \"0c9e4ecb-e4fb-4d66-99a4-b17cf8f4c0cc\") " pod="openshift-multus/multus-52x9m" Oct 30 19:17:07.284474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.283211 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b70be47d-c307-4f47-b075-89a53633411d-etc-sysctl-conf\") pod \"tuned-8r4wz\" (UID: \"b70be47d-c307-4f47-b075-89a53633411d\") " pod="openshift-cluster-node-tuning-operator/tuned-8r4wz" Oct 30 19:17:07.673769 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:07.673742 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:09.708804 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:09.708773 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5vdt7" Oct 30 19:17:10.197197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:10.197172 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7646648d5d-jd8l2" Oct 30 19:17:10.209135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:10.209107 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:10.368296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:10.368271 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:17.324900 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:17.324827 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:17.325340 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:17.325095 6511 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:17:20.195759 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:20.195730 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hcv7r" Oct 30 19:17:24.375738 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:24.375704 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:17:25.052882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:25.052854 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nbd9r" Oct 30 19:17:38.149007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.148976 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9hzhx" Oct 30 19:17:38.655321 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655291 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:17:38.655687 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655656 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="prometheus" containerID="cri-o://f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" gracePeriod=600 Oct 30 19:17:38.655858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655696 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="thanos-sidecar" containerID="cri-o://c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" gracePeriod=600 Oct 30 19:17:38.655858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655718 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" gracePeriod=600 Oct 30 19:17:38.655858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655733 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-web" containerID="cri-o://dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" gracePeriod=600 Oct 30 19:17:38.655858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655759 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="config-reloader" containerID="cri-o://40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" gracePeriod=600 Oct 30 19:17:38.655858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.655692 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy" containerID="cri-o://be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" gracePeriod=600 Oct 30 19:17:38.905520 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:38.905450 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.013788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013755 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.013788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013791 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013826 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013844 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013878 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013920 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013946 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.013973 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014000 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014059 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014087 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014113 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014141 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014173 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014203 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014241 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014246 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014282 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014315 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014350 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014397 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") pod \"7951e5f1-5cb5-4185-836d-599ac3d2223f\" (UID: \"7951e5f1-5cb5-4185-836d-599ac3d2223f\") " Oct 30 19:17:39.014893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014644 6511 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.014893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.014671 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.015917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.015436 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:39.016225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.016195 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.016891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.016868 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.017305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.017279 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.017398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.017279 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.017398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.017374 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.017750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.017690 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:39.018389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.018362 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:39.018655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.018626 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:39.018766 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.018707 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.018872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.018837 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.018998 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.018972 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.019097 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.019048 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.019257 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.019231 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.019597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.019565 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "kube-api-access-xxc8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:39.020004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.019974 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.020088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.020018 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.029367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.029347 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config" pod "7951e5f1-5cb5-4185-836d-599ac3d2223f" (UID: "7951e5f1-5cb5-4185-836d-599ac3d2223f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:39.115160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115127 6511 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115156 6511 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115172 6511 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115186 6511 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-metrics-client-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115198 6511 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-db\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115211 6511 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-grpc-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115222 6511 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-config\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115234 6511 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7951e5f1-5cb5-4185-836d-599ac3d2223f-config-out\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115246 6511 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115259 6511 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-tls-assets\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115307 6511 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-kube-rbac-proxy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115322 6511 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115335 6511 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115349 6511 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-web-config\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115364 6511 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115380 6511 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115825 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115394 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxc8v\" (UniqueName: \"kubernetes.io/projected/7951e5f1-5cb5-4185-836d-599ac3d2223f-kube-api-access-xxc8v\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115825 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115407 6511 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7951e5f1-5cb5-4185-836d-599ac3d2223f-configmap-metrics-client-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.115825 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.115451 6511 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7951e5f1-5cb5-4185-836d-599ac3d2223f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.247373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247288 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" exitCode=0 Oct 30 19:17:39.247373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247314 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" exitCode=0 Oct 30 19:17:39.247373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247323 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" exitCode=0 Oct 30 19:17:39.247373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247330 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" exitCode=0 Oct 30 19:17:39.247373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247336 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" exitCode=0 Oct 30 19:17:39.247373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247343 6511 generic.go:356] "Generic (PLEG): container finished" podID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" exitCode=0 Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247377 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247377 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247477 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247522 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247546 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247564 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247604 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247644 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} Oct 30 19:17:39.247981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.247688 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7951e5f1-5cb5-4185-836d-599ac3d2223f","Type":"ContainerDied","Data":"2ce72a8106dec1217bf4047ea5e8e0a3a224990934fc80841eb5a4909555f355"} Oct 30 19:17:39.255622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.255607 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.262203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.262183 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.268577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.268559 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.272550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.272525 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:17:39.275666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.275651 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.278213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.278195 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:17:39.282577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.282559 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.289628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.289611 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.295708 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.295692 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.295971 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.295953 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.296033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.295981 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} err="failed to get container status \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" Oct 30 19:17:39.296033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296011 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.296273 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.296254 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.296314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296279 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} err="failed to get container status \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" Oct 30 19:17:39.296314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296294 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.296561 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.296544 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.296598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296565 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} err="failed to get container status \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" Oct 30 19:17:39.296598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296578 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.296818 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.296800 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.296880 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296830 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} err="failed to get container status \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" Oct 30 19:17:39.296880 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.296850 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.297092 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.297075 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.297145 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297096 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} err="failed to get container status \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" Oct 30 19:17:39.297145 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297109 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.297316 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.297299 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.297377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297325 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} err="failed to get container status \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" Oct 30 19:17:39.297377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297346 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.297574 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:39.297558 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.297622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297578 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} err="failed to get container status \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" Oct 30 19:17:39.297622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297591 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.297799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297780 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} err="failed to get container status \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" Oct 30 19:17:39.297842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297800 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.298015 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.297997 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} err="failed to get container status \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" Oct 30 19:17:39.298075 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298015 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.298226 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298208 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} err="failed to get container status \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" Oct 30 19:17:39.298281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298226 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.298483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298465 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} err="failed to get container status \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" Oct 30 19:17:39.298537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298483 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.298689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298673 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} err="failed to get container status \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" Oct 30 19:17:39.298689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298689 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.298876 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298859 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} err="failed to get container status \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" Oct 30 19:17:39.298918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.298876 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.299080 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299054 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} err="failed to get container status \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" Oct 30 19:17:39.299152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299079 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.299283 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299265 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} err="failed to get container status \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" Oct 30 19:17:39.299323 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299283 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.299490 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299472 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} err="failed to get container status \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" Oct 30 19:17:39.299565 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299492 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.299710 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299695 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} err="failed to get container status \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" Oct 30 19:17:39.299757 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299710 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.299909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299891 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} err="failed to get container status \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" Oct 30 19:17:39.299953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.299910 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.300108 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300093 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} err="failed to get container status \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" Oct 30 19:17:39.300149 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300108 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.300299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300280 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} err="failed to get container status \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" Oct 30 19:17:39.300348 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300300 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.300519 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300502 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} err="failed to get container status \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" Oct 30 19:17:39.300569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300519 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.300765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300748 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} err="failed to get container status \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" Oct 30 19:17:39.300765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300764 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.300983 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300965 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} err="failed to get container status \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" Oct 30 19:17:39.301023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.300984 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.301172 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301156 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} err="failed to get container status \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" Oct 30 19:17:39.301172 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301170 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.301334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301321 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} err="failed to get container status \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" Oct 30 19:17:39.301334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301334 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.301573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301557 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} err="failed to get container status \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" Oct 30 19:17:39.301573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301573 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.301775 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301758 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} err="failed to get container status \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" Oct 30 19:17:39.301815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301775 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.301964 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301949 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} err="failed to get container status \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" Oct 30 19:17:39.302005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.301964 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.302162 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302140 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} err="failed to get container status \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" Oct 30 19:17:39.302203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302163 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.302360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302342 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} err="failed to get container status \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" Oct 30 19:17:39.302397 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302361 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.302587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302570 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} err="failed to get container status \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" Oct 30 19:17:39.302641 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302587 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.302832 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302810 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} err="failed to get container status \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" Oct 30 19:17:39.302906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.302832 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.303057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303026 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} err="failed to get container status \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" Oct 30 19:17:39.303057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303053 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.303437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303398 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} err="failed to get container status \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" Oct 30 19:17:39.303536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303440 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.303675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303658 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} err="failed to get container status \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" Oct 30 19:17:39.303718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303675 6511 scope.go:122] "RemoveContainer" containerID="4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6" Oct 30 19:17:39.303911 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303895 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6"} err="failed to get container status \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": rpc error: code = NotFound desc = could not find container \"4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6\": container with ID starting with 4a3f666b61303d479aff26a25517ffbc9e88c54eb692e4ce752ba4cc8ae921a6 not found: ID does not exist" Oct 30 19:17:39.303950 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.303912 6511 scope.go:122] "RemoveContainer" containerID="be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5" Oct 30 19:17:39.304119 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304102 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5"} err="failed to get container status \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": rpc error: code = NotFound desc = could not find container \"be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5\": container with ID starting with be0ba419fb1c759c018e8e10aa1b62dc111ab74323b2c6687864c4d6a51563b5 not found: ID does not exist" Oct 30 19:17:39.304162 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304122 6511 scope.go:122] "RemoveContainer" containerID="dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d" Oct 30 19:17:39.304367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304350 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d"} err="failed to get container status \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": rpc error: code = NotFound desc = could not find container \"dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d\": container with ID starting with dc248ec147c7828199d401da19ea757550426d1460aafaa50ae60e776e1e9b1d not found: ID does not exist" Oct 30 19:17:39.304426 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304368 6511 scope.go:122] "RemoveContainer" containerID="c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e" Oct 30 19:17:39.304606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304589 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e"} err="failed to get container status \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": rpc error: code = NotFound desc = could not find container \"c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e\": container with ID starting with c6db74d8bf36537679d44edc793e3c36285fb7366a0a4607965c21c2d469e19e not found: ID does not exist" Oct 30 19:17:39.304640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304607 6511 scope.go:122] "RemoveContainer" containerID="40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb" Oct 30 19:17:39.304811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304796 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb"} err="failed to get container status \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": rpc error: code = NotFound desc = could not find container \"40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb\": container with ID starting with 40324fe033616da96351819080e7dc8735dcb1699bc70ffdfe854fb11fd78bfb not found: ID does not exist" Oct 30 19:17:39.304854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.304812 6511 scope.go:122] "RemoveContainer" containerID="f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac" Oct 30 19:17:39.305022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.305005 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac"} err="failed to get container status \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": rpc error: code = NotFound desc = could not find container \"f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac\": container with ID starting with f336c128197c248a511f6e08cb0795f9a6fa58292e7215b62e7fe5e7d9d320ac not found: ID does not exist" Oct 30 19:17:39.305058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.305022 6511 scope.go:122] "RemoveContainer" containerID="8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d" Oct 30 19:17:39.305228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.305208 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d"} err="failed to get container status \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": rpc error: code = NotFound desc = could not find container \"8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d\": container with ID starting with 8a3ee5d3a56bfdbd4fc11f69b7b73e3c06b6af0bf456d37265a7b5f37b79247d not found: ID does not exist" Oct 30 19:17:39.317190 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317169 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:17:39.317522 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317506 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-thanos" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317524 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-thanos" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317535 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="config-reloader" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317540 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="config-reloader" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317550 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="prometheus" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317555 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="prometheus" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317562 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317567 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317576 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-web" Oct 30 19:17:39.317577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317580 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-web" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317589 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="thanos-sidecar" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317593 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="thanos-sidecar" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317598 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="init-config-reloader" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317603 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="init-config-reloader" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317645 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-thanos" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317652 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="config-reloader" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317658 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="prometheus" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317663 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317669 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="kube-rbac-proxy-web" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317675 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="thanos-sidecar" Oct 30 19:17:39.317860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.317680 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" containerName="init-config-reloader" Oct 30 19:17:39.320896 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.320882 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.328245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.328222 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:17:39.328562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.328545 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:17:39.332159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.332138 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:17:39.332489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.332476 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:17:39.332553 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.332525 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:17:39.337948 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.337931 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:17:39.339001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.338984 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:17:39.339304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339290 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:17:39.339578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339560 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:17:39.339640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339576 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:17:39.339640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339593 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9rd2gbba2224\"" Oct 30 19:17:39.339640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339603 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:17:39.339640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339615 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:17:39.339640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339615 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:17:39.339640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339618 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:17:39.339885 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339769 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rklsm\"" Oct 30 19:17:39.339885 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.339779 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:17:39.350136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.350069 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:17:39.417537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417509 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417549 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-config\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417577 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417624 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417793 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417739 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417793 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417763 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417793 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417788 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2828c894-def5-44ea-91ba-b3560cd7419f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417833 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417868 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.417906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417887 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klmdn\" (UniqueName: \"kubernetes.io/projected/2828c894-def5-44ea-91ba-b3560cd7419f-kube-api-access-klmdn\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417935 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.417972 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418007 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418029 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418048 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418068 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2828c894-def5-44ea-91ba-b3560cd7419f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418089 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418111 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418180 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.418304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.418200 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.518835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518761 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2828c894-def5-44ea-91ba-b3560cd7419f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.518835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518803 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.518835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518828 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518854 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klmdn\" (UniqueName: \"kubernetes.io/projected/2828c894-def5-44ea-91ba-b3560cd7419f-kube-api-access-klmdn\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518880 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518901 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518929 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.518958 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519165 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519207 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2828c894-def5-44ea-91ba-b3560cd7419f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519206 6511 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:17:39.519564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519470 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519505 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519539 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519566 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519599 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519627 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-config\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519653 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519676 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519937 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519717 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.519937 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519741 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.520059 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.519998 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.520270 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.520242 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.520456 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.520248 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.520568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.520524 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.520772 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.520743 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.520996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.520972 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.522380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.522353 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.522547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.522499 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2828c894-def5-44ea-91ba-b3560cd7419f-config-out\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.522744 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.522691 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.523234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.523207 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.523313 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.523253 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2828c894-def5-44ea-91ba-b3560cd7419f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.523733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.523705 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.524020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.523965 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.524118 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.524018 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2828c894-def5-44ea-91ba-b3560cd7419f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.524682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.524655 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.524897 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.524878 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-web-config\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.525071 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.525055 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.525395 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.525372 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.525929 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.525915 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2828c894-def5-44ea-91ba-b3560cd7419f-config\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.532205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.532183 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klmdn\" (UniqueName: \"kubernetes.io/projected/2828c894-def5-44ea-91ba-b3560cd7419f-kube-api-access-klmdn\") pod \"prometheus-k8s-0\" (UID: \"2828c894-def5-44ea-91ba-b3560cd7419f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.641075 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.641052 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:39.920519 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:39.920148 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:17:39.922925 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:17:39.922894 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2828c894_def5_44ea_91ba_b3560cd7419f.slice/crio-9fcf465f1748668de1b0c81b9f2a0afd804040efa3a6b99c90f79b0bd0fd1ea7 WatchSource:0}: Error finding container 9fcf465f1748668de1b0c81b9f2a0afd804040efa3a6b99c90f79b0bd0fd1ea7: Status 404 returned error can't find the container with id 9fcf465f1748668de1b0c81b9f2a0afd804040efa3a6b99c90f79b0bd0fd1ea7 Oct 30 19:17:40.093308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:40.093279 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7951e5f1-5cb5-4185-836d-599ac3d2223f" path="/var/lib/kubelet/pods/7951e5f1-5cb5-4185-836d-599ac3d2223f/volumes" Oct 30 19:17:40.252344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:40.252271 6511 generic.go:356] "Generic (PLEG): container finished" podID="2828c894-def5-44ea-91ba-b3560cd7419f" containerID="c35edbbd2b566d7b66f7e2b2029cc51be281d950fe9dc20bae18db24a80b95c7" exitCode=0 Oct 30 19:17:40.252783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:40.252360 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerDied","Data":"c35edbbd2b566d7b66f7e2b2029cc51be281d950fe9dc20bae18db24a80b95c7"} Oct 30 19:17:40.252783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:40.252460 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"9fcf465f1748668de1b0c81b9f2a0afd804040efa3a6b99c90f79b0bd0fd1ea7"} Oct 30 19:17:41.260337 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.260302 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"98a38afbb2e30b3bdb10b592a19c2a1a0fd28e7fd68e6bf47c61f7bcf2ab9fdb"} Oct 30 19:17:41.260733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.260394 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"7fdc6a4e69285df17dce5388c1128e2ed84b1735eb8c854c5a1b45dbf50a06ba"} Oct 30 19:17:41.260733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.260439 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"54320ec5b1bf4631b0ad9399dcc00bc83dbe232e22caff56225a39efaaf7c5c6"} Oct 30 19:17:41.260733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.260464 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"ddb9e2d9831fd321993101d69194880322dd551530b0d3c3583d22fcf4fbc4a7"} Oct 30 19:17:41.260733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.260489 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"9984f527cd601281ddbfa037d0db415487f22158e474e76668e84680cb1f5545"} Oct 30 19:17:41.260733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.260511 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2828c894-def5-44ea-91ba-b3560cd7419f","Type":"ContainerStarted","Data":"3c0bc1639bf10d254a709fc0ab8d4d6403822640c834fc295a3c657f2c05e9a9"} Oct 30 19:17:41.291604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:41.291552 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.275407861 podStartE2EDuration="2.291539114s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.131253ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:41.289821277 +0000 UTC m=+35.313425172" watchObservedRunningTime="2026-10-30 19:17:41.291539114 +0000 UTC m=+35.315143013" Oct 30 19:17:44.641288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:17:44.641250 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:17:54.376280 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:17:54.376245 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:18:04.904828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:18:04.904793 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:18:04.921263 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:18:04.921237 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:18:05.357839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:18:05.357809 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:18:24.376549 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:18:24.376513 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:18:54.376605 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:18:54.376528 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:19:24.376640 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:19:24.376604 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:19:54.376778 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:19:54.376744 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:19:55.126325 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.126290 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp"] Oct 30 19:19:55.129760 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.129737 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.135405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.135380 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-b5s2f\"" Oct 30 19:19:55.135542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.135466 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.135542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.135485 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.146104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.146079 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp"] Oct 30 19:19:55.204922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.204880 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.205060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.204927 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vv7\" (UniqueName: \"kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.205060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.204956 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.305292 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.305262 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.305450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.305297 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vv7\" (UniqueName: \"kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.305450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.305316 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.305688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.305670 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.305745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.305729 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.315265 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.315246 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vv7\" (UniqueName: \"kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.446039 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.445974 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:19:55.717222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.717178 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp"] Oct 30 19:19:55.719388 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:55.719363 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e07d528_36c8_42bf_b8b7_bcb7196dbb9d.slice/crio-6132e27372f024ad3f968bb678cb2b8c1a0ff1484ebc82dcd3b9997f49a6b44f WatchSource:0}: Error finding container 6132e27372f024ad3f968bb678cb2b8c1a0ff1484ebc82dcd3b9997f49a6b44f: Status 404 returned error can't find the container with id 6132e27372f024ad3f968bb678cb2b8c1a0ff1484ebc82dcd3b9997f49a6b44f Oct 30 19:19:55.721352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:55.721328 6511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:19:56.227228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.227195 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q"] Oct 30 19:19:56.230857 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.230832 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.236824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.236551 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-hcktp\"" Oct 30 19:19:56.241201 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.241180 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q"] Oct 30 19:19:56.312778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.312748 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.313001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.312786 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8ff\" (UniqueName: \"kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.313001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.312923 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.323456 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.323435 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7"] Oct 30 19:19:56.326730 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.326708 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.332716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.332694 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-smlmc\"" Oct 30 19:19:56.339161 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.339102 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7"] Oct 30 19:19:56.413577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.413550 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.413743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.413603 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.413743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.413678 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.413856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.413796 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.413856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.413831 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8ff\" (UniqueName: \"kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.413856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.413851 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd55s\" (UniqueName: \"kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.414096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.414076 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.414154 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.414113 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.429761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.429736 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8ff\" (UniqueName: \"kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.431158 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.431140 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg"] Oct 30 19:19:56.434866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.434849 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.439862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.439841 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-tkgr4\"" Oct 30 19:19:56.444435 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.444389 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg"] Oct 30 19:19:56.514777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.514714 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.514777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.514761 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.515218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.514842 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd55s\" (UniqueName: \"kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.515218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.514891 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.515218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.514952 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lzv\" (UniqueName: \"kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.515218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.515105 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.515218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.515173 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.515592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.515569 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.517855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.517833 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7"] Oct 30 19:19:56.521247 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.521228 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.526092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.526069 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-r2kp8\"" Oct 30 19:19:56.527548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.527527 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd55s\" (UniqueName: \"kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.532291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.532232 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7"] Oct 30 19:19:56.550890 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.550872 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:19:56.615995 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.615961 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drckw\" (UniqueName: \"kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.616120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616068 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.616191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616120 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.616191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616156 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lzv\" (UniqueName: \"kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.616309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616294 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.616366 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616334 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.616498 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616471 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.616667 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.616643 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.632443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.625491 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lzv\" (UniqueName: \"kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.644334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.644310 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:19:56.700001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.697850 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" event={"ID":"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d","Type":"ContainerStarted","Data":"6132e27372f024ad3f968bb678cb2b8c1a0ff1484ebc82dcd3b9997f49a6b44f"} Oct 30 19:19:56.716887 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.716844 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drckw\" (UniqueName: \"kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.717040 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.716930 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.717098 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.717077 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.717570 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.717547 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.717824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.717777 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.728350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.728328 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drckw\" (UniqueName: \"kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.756142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.756119 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:19:56.830025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.829998 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q"] Oct 30 19:19:56.836948 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:56.836917 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a4e301_db40_4d07_9685_0a6a9eeb92b7.slice/crio-9a9708ee8dc6a19971157c1aa2e5f1ac7b20994fa142d7ce203e582217fcb8bc WatchSource:0}: Error finding container 9a9708ee8dc6a19971157c1aa2e5f1ac7b20994fa142d7ce203e582217fcb8bc: Status 404 returned error can't find the container with id 9a9708ee8dc6a19971157c1aa2e5f1ac7b20994fa142d7ce203e582217fcb8bc Oct 30 19:19:56.849840 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.849815 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:19:56.933558 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:56.933529 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7"] Oct 30 19:19:56.937357 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:56.937329 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eeeed4_ee5d_46ef_8792_b0a3f401abbc.slice/crio-7bdc0cd548716e3f1136f10969d8262524f5ad37a7c184bb713fb16db045fd41 WatchSource:0}: Error finding container 7bdc0cd548716e3f1136f10969d8262524f5ad37a7c184bb713fb16db045fd41: Status 404 returned error can't find the container with id 7bdc0cd548716e3f1136f10969d8262524f5ad37a7c184bb713fb16db045fd41 Oct 30 19:19:57.038508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:57.038425 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg"] Oct 30 19:19:57.042095 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:57.042068 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1be0d18_cd7e_4d9e_a79b_0b7d4f8e1dcf.slice/crio-eda063b8da7c2c4f52e1cc443cbc11f44a7b0fb368f47a10137880926072f0fc WatchSource:0}: Error finding container eda063b8da7c2c4f52e1cc443cbc11f44a7b0fb368f47a10137880926072f0fc: Status 404 returned error can't find the container with id eda063b8da7c2c4f52e1cc443cbc11f44a7b0fb368f47a10137880926072f0fc Oct 30 19:19:57.133002 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:57.132973 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7"] Oct 30 19:19:57.136143 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:57.136115 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf100c619_9472_45d5_8e54_146a29da5d19.slice/crio-22e463a239db36e59d098d6d94809601f3da3544ea06f1ec784c80b6e0f6a46e WatchSource:0}: Error finding container 22e463a239db36e59d098d6d94809601f3da3544ea06f1ec784c80b6e0f6a46e: Status 404 returned error can't find the container with id 22e463a239db36e59d098d6d94809601f3da3544ea06f1ec784c80b6e0f6a46e Oct 30 19:19:57.705607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:57.705485 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" event={"ID":"14a4e301-db40-4d07-9685-0a6a9eeb92b7","Type":"ContainerStarted","Data":"9a9708ee8dc6a19971157c1aa2e5f1ac7b20994fa142d7ce203e582217fcb8bc"} Oct 30 19:19:57.721914 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:57.721519 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" event={"ID":"f100c619-9472-45d5-8e54-146a29da5d19","Type":"ContainerStarted","Data":"22e463a239db36e59d098d6d94809601f3da3544ea06f1ec784c80b6e0f6a46e"} Oct 30 19:19:57.724746 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:57.724680 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" event={"ID":"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf","Type":"ContainerStarted","Data":"eda063b8da7c2c4f52e1cc443cbc11f44a7b0fb368f47a10137880926072f0fc"} Oct 30 19:19:57.726503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:57.726441 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" event={"ID":"56eeeed4-ee5d-46ef-8792-b0a3f401abbc","Type":"ContainerStarted","Data":"7bdc0cd548716e3f1136f10969d8262524f5ad37a7c184bb713fb16db045fd41"} Oct 30 19:19:58.492054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.491805 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb"] Oct 30 19:19:58.495846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.495821 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.505594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.505571 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:19:58.507787 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.507768 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:19:58.508341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.508317 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gt4zp\"" Oct 30 19:19:58.509148 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.509127 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb"] Oct 30 19:19:58.513801 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.513778 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff"] Oct 30 19:19:58.524141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.524113 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.532672 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.532647 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.532788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.532709 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.532788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.532770 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.532909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.532877 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.532959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.532910 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.533016 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.532977 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.533072 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533022 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.533128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533104 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29kvv\" (UniqueName: \"kubernetes.io/projected/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-kube-api-access-29kvv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.533180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533138 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.533180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533165 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.533279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533214 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.533279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533254 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdpb\" (UniqueName: \"kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.533486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.533464 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-b5vtd\"" Oct 30 19:19:58.534357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.534333 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff"] Oct 30 19:19:58.605703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.605675 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:19:58.634035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634001 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634050 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.634151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634085 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634112 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634137 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634169 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634232 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29kvv\" (UniqueName: \"kubernetes.io/projected/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-kube-api-access-29kvv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634250 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634268 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634293 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634311 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdpb\" (UniqueName: \"kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.634336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634332 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.634787 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.634764 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.635432 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.635385 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.636377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.636319 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.636587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.636566 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.637058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.637027 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.637312 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.637293 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.637572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.637552 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.640687 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.640661 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.643174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.643092 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.650357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.650329 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdpb\" (UniqueName: \"kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:58.662838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.662806 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.678395 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.678371 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29kvv\" (UniqueName: \"kubernetes.io/projected/ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f-kube-api-access-29kvv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-x2wnb\" (UID: \"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.833045 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.833008 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:19:58.857106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:58.854679 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:19:59.142298 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:59.142216 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb"] Oct 30 19:19:59.145515 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:59.145474 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3a8e75_ee0e_4ba7_8aaa_ad3da51a227f.slice/crio-600b243638e5464f8b61847bbcc19dda7087e4c6c9ac413c70ef9d4768f4a1cd WatchSource:0}: Error finding container 600b243638e5464f8b61847bbcc19dda7087e4c6c9ac413c70ef9d4768f4a1cd: Status 404 returned error can't find the container with id 600b243638e5464f8b61847bbcc19dda7087e4c6c9ac413c70ef9d4768f4a1cd Oct 30 19:19:59.175115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:59.175036 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff"] Oct 30 19:19:59.179940 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:19:59.179912 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode651d6b1_8f52_457e_b6cd_63517553ac6d.slice/crio-a3208207ed9be92899ed2d162604052c87efa47cf5f46d7928221a8b7815c8d9 WatchSource:0}: Error finding container a3208207ed9be92899ed2d162604052c87efa47cf5f46d7928221a8b7815c8d9: Status 404 returned error can't find the container with id a3208207ed9be92899ed2d162604052c87efa47cf5f46d7928221a8b7815c8d9 Oct 30 19:19:59.746149 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:59.746109 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" event={"ID":"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f","Type":"ContainerStarted","Data":"600b243638e5464f8b61847bbcc19dda7087e4c6c9ac413c70ef9d4768f4a1cd"} Oct 30 19:19:59.750978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:19:59.750948 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" event={"ID":"e651d6b1-8f52-457e-b6cd-63517553ac6d","Type":"ContainerStarted","Data":"a3208207ed9be92899ed2d162604052c87efa47cf5f46d7928221a8b7815c8d9"} Oct 30 19:20:05.680531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:05.680489 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:20:05.680995 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:05.680588 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:20:05.680995 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:05.680631 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:20:05.808200 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:05.807310 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" event={"ID":"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f","Type":"ContainerStarted","Data":"599c2c4487772ce097deaafaf62f1f079e320a5c53bfa22464cbbf3a62474b9d"} Oct 30 19:20:05.840400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:05.839972 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:20:06.840341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:06.840308 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:20:06.861238 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:06.861183 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podStartSLOduration=2.330768412 podStartE2EDuration="8.861166154s" totalImagesPullingTime="6.530397742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:05.843097479 +0000 UTC m=+179.866701383" watchObservedRunningTime="2026-10-30 19:20:06.861166154 +0000 UTC m=+180.884770051" Oct 30 19:20:07.831110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:07.831012 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerID="1a65851e02c1446b838ab2b7e3375671c5a98f471a56d7c1189a59e794fada37" exitCode=0 Oct 30 19:20:07.833102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:07.832672 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" event={"ID":"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d","Type":"ContainerDied","Data":"1a65851e02c1446b838ab2b7e3375671c5a98f471a56d7c1189a59e794fada37"} Oct 30 19:20:07.833102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:07.832721 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:20:07.834013 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:07.833983 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:20:08.839046 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:08.839007 6511 generic.go:356] "Generic (PLEG): container finished" podID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerID="71092cfcd08f3e5504110a14c2c8011f09715b513a22b7fe5c74e184a1950f1a" exitCode=0 Oct 30 19:20:08.840071 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:08.840049 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" event={"ID":"e651d6b1-8f52-457e-b6cd-63517553ac6d","Type":"ContainerDied","Data":"71092cfcd08f3e5504110a14c2c8011f09715b513a22b7fe5c74e184a1950f1a"} Oct 30 19:20:09.844171 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:09.844099 6511 generic.go:356] "Generic (PLEG): container finished" podID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerID="c429744c00fcdfa3914a44b85c756fc259e170bdcf824ebf469126a21c775c44" exitCode=0 Oct 30 19:20:09.844587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:09.844186 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" event={"ID":"56eeeed4-ee5d-46ef-8792-b0a3f401abbc","Type":"ContainerDied","Data":"c429744c00fcdfa3914a44b85c756fc259e170bdcf824ebf469126a21c775c44"} Oct 30 19:20:09.846542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:09.845879 6511 generic.go:356] "Generic (PLEG): container finished" podID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerID="308f36db9b975df60f735d2c447287d4756bd16c8ef48a5d7a44a4473af4d074" exitCode=0 Oct 30 19:20:09.846542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:09.846044 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" event={"ID":"14a4e301-db40-4d07-9685-0a6a9eeb92b7","Type":"ContainerDied","Data":"308f36db9b975df60f735d2c447287d4756bd16c8ef48a5d7a44a4473af4d074"} Oct 30 19:20:10.856063 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:10.856028 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerID="f926e335f8db021a73253fdb5a7d611ada1b0e6f925ff64163e10e6c01689197" exitCode=0 Oct 30 19:20:10.856500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:10.856102 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" event={"ID":"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d","Type":"ContainerDied","Data":"f926e335f8db021a73253fdb5a7d611ada1b0e6f925ff64163e10e6c01689197"} Oct 30 19:20:10.857784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:10.857751 6511 generic.go:356] "Generic (PLEG): container finished" podID="f100c619-9472-45d5-8e54-146a29da5d19" containerID="efe0be89a8a33ba25c47af6e245566b6add9e8659ce32149e9ed941e21f7abc4" exitCode=0 Oct 30 19:20:10.857897 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:10.857824 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" event={"ID":"f100c619-9472-45d5-8e54-146a29da5d19","Type":"ContainerDied","Data":"efe0be89a8a33ba25c47af6e245566b6add9e8659ce32149e9ed941e21f7abc4"} Oct 30 19:20:10.859477 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:10.859452 6511 generic.go:356] "Generic (PLEG): container finished" podID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerID="32f74a1c16463401a24614a5e09d80ab84e987a43b0b0b31abbefde7ad8c749f" exitCode=0 Oct 30 19:20:10.859570 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:10.859487 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" event={"ID":"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf","Type":"ContainerDied","Data":"32f74a1c16463401a24614a5e09d80ab84e987a43b0b0b31abbefde7ad8c749f"} Oct 30 19:20:11.865438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.865391 6511 generic.go:356] "Generic (PLEG): container finished" podID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerID="c245c50fac89d09d46f88aa1fef217af713828d2a90710cb95f6723952e5299e" exitCode=0 Oct 30 19:20:11.865883 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.865459 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" event={"ID":"14a4e301-db40-4d07-9685-0a6a9eeb92b7","Type":"ContainerDied","Data":"c245c50fac89d09d46f88aa1fef217af713828d2a90710cb95f6723952e5299e"} Oct 30 19:20:11.868436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.868390 6511 generic.go:356] "Generic (PLEG): container finished" podID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerID="c36e96b041079ad86f1632888dc34ff8a4228375766aa003d57b83cc83251c97" exitCode=0 Oct 30 19:20:11.868537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.868478 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" event={"ID":"e651d6b1-8f52-457e-b6cd-63517553ac6d","Type":"ContainerDied","Data":"c36e96b041079ad86f1632888dc34ff8a4228375766aa003d57b83cc83251c97"} Oct 30 19:20:11.871085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.871062 6511 generic.go:356] "Generic (PLEG): container finished" podID="f100c619-9472-45d5-8e54-146a29da5d19" containerID="1e64016baa775206b0977cd0ff0a138362f8cb03112eb4c2b32f90c78f87b42a" exitCode=0 Oct 30 19:20:11.871174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.871156 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" event={"ID":"f100c619-9472-45d5-8e54-146a29da5d19","Type":"ContainerDied","Data":"1e64016baa775206b0977cd0ff0a138362f8cb03112eb4c2b32f90c78f87b42a"} Oct 30 19:20:11.873457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.873435 6511 generic.go:356] "Generic (PLEG): container finished" podID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerID="3a90188726d48087a0b35d00855aadb67c3ac231a240b70ee63f74dd16289f48" exitCode=0 Oct 30 19:20:11.873558 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.873524 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" event={"ID":"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf","Type":"ContainerDied","Data":"3a90188726d48087a0b35d00855aadb67c3ac231a240b70ee63f74dd16289f48"} Oct 30 19:20:11.875541 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.875523 6511 generic.go:356] "Generic (PLEG): container finished" podID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerID="0a8ff61072ef2490ecd6ab5ab9840023e520d055972a86e8372d206aa38d5619" exitCode=0 Oct 30 19:20:11.875643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:11.875551 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" event={"ID":"56eeeed4-ee5d-46ef-8792-b0a3f401abbc","Type":"ContainerDied","Data":"0a8ff61072ef2490ecd6ab5ab9840023e520d055972a86e8372d206aa38d5619"} Oct 30 19:20:21.743122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.741173 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc"] Oct 30 19:20:21.811974 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.811177 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc"] Oct 30 19:20:21.811974 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.811308 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.863516 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.863475 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:20:21.864102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.863537 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:20:21.864513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.864488 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:20:21.865452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.865401 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:20:21.865597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.865578 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:20:21.867434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.867216 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hbls9\"" Oct 30 19:20:21.873805 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.873780 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.873918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.873832 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.873982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.873914 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46143cf4-b608-4435-9588-d265ebe5f5ab-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.873982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.873966 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgg8k\" (UniqueName: \"kubernetes.io/projected/46143cf4-b608-4435-9588-d265ebe5f5ab-kube-api-access-bgg8k\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.874076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.874031 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46143cf4-b608-4435-9588-d265ebe5f5ab-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.874343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.874287 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.975357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.975316 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46143cf4-b608-4435-9588-d265ebe5f5ab-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.975559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.975372 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.975559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.975437 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.975559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.975475 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.975559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.975506 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46143cf4-b608-4435-9588-d265ebe5f5ab-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.975559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.975555 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgg8k\" (UniqueName: \"kubernetes.io/projected/46143cf4-b608-4435-9588-d265ebe5f5ab-kube-api-access-bgg8k\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.976228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.976198 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46143cf4-b608-4435-9588-d265ebe5f5ab-manifests\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.994747 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.994687 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.994747 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.994698 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-metrics-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.995055 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.994975 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46143cf4-b608-4435-9588-d265ebe5f5ab-tmp\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.995632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.995586 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgg8k\" (UniqueName: \"kubernetes.io/projected/46143cf4-b608-4435-9588-d265ebe5f5ab-kube-api-access-bgg8k\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:21.996305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:21.996285 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46143cf4-b608-4435-9588-d265ebe5f5ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fb5c5984-qhdzc\" (UID: \"46143cf4-b608-4435-9588-d265ebe5f5ab\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:22.174817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:22.174533 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:22.564366 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:22.564339 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc"] Oct 30 19:20:22.565945 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:20:22.565910 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46143cf4_b608_4435_9588_d265ebe5f5ab.slice/crio-4a5840ebc781ec1861684ef99a403f9d1e9de3711c9b2eae0035ad20fdba1b6f WatchSource:0}: Error finding container 4a5840ebc781ec1861684ef99a403f9d1e9de3711c9b2eae0035ad20fdba1b6f: Status 404 returned error can't find the container with id 4a5840ebc781ec1861684ef99a403f9d1e9de3711c9b2eae0035ad20fdba1b6f Oct 30 19:20:22.955487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:22.955325 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" event={"ID":"46143cf4-b608-4435-9588-d265ebe5f5ab","Type":"ContainerStarted","Data":"4a5840ebc781ec1861684ef99a403f9d1e9de3711c9b2eae0035ad20fdba1b6f"} Oct 30 19:20:23.966725 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:23.966643 6511 generic.go:356] "Generic (PLEG): container finished" podID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerID="c7eff09d122ff14fb025962f7872ab8341bdfa824074e53bf33c77715d6281bb" exitCode=0 Oct 30 19:20:23.967216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:23.966735 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" event={"ID":"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d","Type":"ContainerDied","Data":"c7eff09d122ff14fb025962f7872ab8341bdfa824074e53bf33c77715d6281bb"} Oct 30 19:20:24.399305 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:20:24.399258 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:20:25.179604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.179314 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:20:25.310953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.310921 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util\") pod \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " Oct 30 19:20:25.311139 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.311001 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7\" (UniqueName: \"kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7\") pod \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " Oct 30 19:20:25.311139 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.311048 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle\") pod \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\" (UID: \"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d\") " Oct 30 19:20:25.311916 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.311853 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle" pod "0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" (UID: "0e07d528-36c8-42bf-b8b7-bcb7196dbb9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:25.315001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.314970 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7" pod "0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" (UID: "0e07d528-36c8-42bf-b8b7-bcb7196dbb9d"). InnerVolumeSpecName "kube-api-access-q9vv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:25.316781 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.316756 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util" pod "0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" (UID: "0e07d528-36c8-42bf-b8b7-bcb7196dbb9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:25.412275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.412184 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vv7\" (UniqueName: \"kubernetes.io/projected/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-kube-api-access-q9vv7\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:25.412275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.412223 6511 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:25.412275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.412239 6511 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e07d528-36c8-42bf-b8b7-bcb7196dbb9d-util\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:25.981311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.981263 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" event={"ID":"0e07d528-36c8-42bf-b8b7-bcb7196dbb9d","Type":"ContainerDied","Data":"6132e27372f024ad3f968bb678cb2b8c1a0ff1484ebc82dcd3b9997f49a6b44f"} Oct 30 19:20:25.981485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.981282 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxc7hp" Oct 30 19:20:25.981485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.981344 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6132e27372f024ad3f968bb678cb2b8c1a0ff1484ebc82dcd3b9997f49a6b44f" Oct 30 19:20:25.983877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.983849 6511 generic.go:356] "Generic (PLEG): container finished" podID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerID="016b874c5b346ed051e6899e2ecb8cfd3244bd3f9cc93f1187f80c774f5ddf2c" exitCode=0 Oct 30 19:20:25.984023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.983954 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" event={"ID":"56eeeed4-ee5d-46ef-8792-b0a3f401abbc","Type":"ContainerDied","Data":"016b874c5b346ed051e6899e2ecb8cfd3244bd3f9cc93f1187f80c774f5ddf2c"} Oct 30 19:20:25.986704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.986676 6511 generic.go:356] "Generic (PLEG): container finished" podID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerID="5ee887643343043af65ffb2880814f52cd94aa2892d46edd14f2c20a1ca0afe2" exitCode=0 Oct 30 19:20:25.986837 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:25.986740 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" event={"ID":"14a4e301-db40-4d07-9685-0a6a9eeb92b7","Type":"ContainerDied","Data":"5ee887643343043af65ffb2880814f52cd94aa2892d46edd14f2c20a1ca0afe2"} Oct 30 19:20:26.993833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:26.993753 6511 generic.go:356] "Generic (PLEG): container finished" podID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerID="612ec760581877d79174b2c39e5076540b517884155f7efa92ab4a9d054f2a12" exitCode=0 Oct 30 19:20:26.993833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:26.993807 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" event={"ID":"e651d6b1-8f52-457e-b6cd-63517553ac6d","Type":"ContainerDied","Data":"612ec760581877d79174b2c39e5076540b517884155f7efa92ab4a9d054f2a12"} Oct 30 19:20:26.996235 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:26.996210 6511 generic.go:356] "Generic (PLEG): container finished" podID="f100c619-9472-45d5-8e54-146a29da5d19" containerID="43283da6b2da354fe2337c4575145d96b207f16c126e0c6705765a11b2fc22e4" exitCode=0 Oct 30 19:20:26.997592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:26.996272 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" event={"ID":"f100c619-9472-45d5-8e54-146a29da5d19","Type":"ContainerDied","Data":"43283da6b2da354fe2337c4575145d96b207f16c126e0c6705765a11b2fc22e4"} Oct 30 19:20:26.999252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:26.998752 6511 generic.go:356] "Generic (PLEG): container finished" podID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerID="fea04b3f172d04ceff706356a1482a9709c06eb1c304ca12b85fa6f7e78aea07" exitCode=0 Oct 30 19:20:26.999252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:26.998854 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" event={"ID":"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf","Type":"ContainerDied","Data":"fea04b3f172d04ceff706356a1482a9709c06eb1c304ca12b85fa6f7e78aea07"} Oct 30 19:20:27.160861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.160832 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:20:27.194713 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.194687 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:20:27.229965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.229932 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff\" (UniqueName: \"kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff\") pod \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " Oct 30 19:20:27.230134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.229992 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util\") pod \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " Oct 30 19:20:27.230192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.230130 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle\") pod \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\" (UID: \"14a4e301-db40-4d07-9685-0a6a9eeb92b7\") " Oct 30 19:20:27.231206 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.231172 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle" pod "14a4e301-db40-4d07-9685-0a6a9eeb92b7" (UID: "14a4e301-db40-4d07-9685-0a6a9eeb92b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:27.233128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.233105 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff" pod "14a4e301-db40-4d07-9685-0a6a9eeb92b7" (UID: "14a4e301-db40-4d07-9685-0a6a9eeb92b7"). InnerVolumeSpecName "kube-api-access-qv8ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:27.237917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.237649 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util" pod "14a4e301-db40-4d07-9685-0a6a9eeb92b7" (UID: "14a4e301-db40-4d07-9685-0a6a9eeb92b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:27.331004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.330978 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle\") pod \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " Oct 30 19:20:27.331175 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.331016 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util\") pod \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " Oct 30 19:20:27.331175 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.331088 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s\" (UniqueName: \"kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s\") pod \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\" (UID: \"56eeeed4-ee5d-46ef-8792-b0a3f401abbc\") " Oct 30 19:20:27.331305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.331277 6511 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:27.331305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.331299 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv8ff\" (UniqueName: \"kubernetes.io/projected/14a4e301-db40-4d07-9685-0a6a9eeb92b7-kube-api-access-qv8ff\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:27.331409 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.331313 6511 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/14a4e301-db40-4d07-9685-0a6a9eeb92b7-util\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:27.331582 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.331558 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle" pod "56eeeed4-ee5d-46ef-8792-b0a3f401abbc" (UID: "56eeeed4-ee5d-46ef-8792-b0a3f401abbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:27.333204 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.333178 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s" pod "56eeeed4-ee5d-46ef-8792-b0a3f401abbc" (UID: "56eeeed4-ee5d-46ef-8792-b0a3f401abbc"). InnerVolumeSpecName "kube-api-access-sd55s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:27.336431 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.336387 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util" pod "56eeeed4-ee5d-46ef-8792-b0a3f401abbc" (UID: "56eeeed4-ee5d-46ef-8792-b0a3f401abbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:27.431867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.431831 6511 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-util\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:27.431867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.431864 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd55s\" (UniqueName: \"kubernetes.io/projected/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-kube-api-access-sd55s\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:27.431867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:27.431874 6511 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/56eeeed4-ee5d-46ef-8792-b0a3f401abbc-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.005746 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.005666 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" Oct 30 19:20:28.006239 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.005665 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdvwt7" event={"ID":"56eeeed4-ee5d-46ef-8792-b0a3f401abbc","Type":"ContainerDied","Data":"7bdc0cd548716e3f1136f10969d8262524f5ad37a7c184bb713fb16db045fd41"} Oct 30 19:20:28.006239 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.005781 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bdc0cd548716e3f1136f10969d8262524f5ad37a7c184bb713fb16db045fd41" Oct 30 19:20:28.007624 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.007603 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" Oct 30 19:20:28.007761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.007600 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6p9d4q" event={"ID":"14a4e301-db40-4d07-9685-0a6a9eeb92b7","Type":"ContainerDied","Data":"9a9708ee8dc6a19971157c1aa2e5f1ac7b20994fa142d7ce203e582217fcb8bc"} Oct 30 19:20:28.007761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.007711 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a9708ee8dc6a19971157c1aa2e5f1ac7b20994fa142d7ce203e582217fcb8bc" Oct 30 19:20:28.009451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.009424 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" event={"ID":"46143cf4-b608-4435-9588-d265ebe5f5ab","Type":"ContainerStarted","Data":"b2283bac81cb9973c7377751d1d625e09cab1de7a8ef2ed060bd9dde854c0f87"} Oct 30 19:20:28.174266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.174246 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:20:28.175267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.175250 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:20:28.181430 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.181397 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:20:28.339312 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339282 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle\") pod \"f100c619-9472-45d5-8e54-146a29da5d19\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " Oct 30 19:20:28.339508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339326 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw\" (UniqueName: \"kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw\") pod \"f100c619-9472-45d5-8e54-146a29da5d19\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " Oct 30 19:20:28.339508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339354 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util\") pod \"f100c619-9472-45d5-8e54-146a29da5d19\" (UID: \"f100c619-9472-45d5-8e54-146a29da5d19\") " Oct 30 19:20:28.339508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339388 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv\" (UniqueName: \"kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv\") pod \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " Oct 30 19:20:28.339508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339442 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb\" (UniqueName: \"kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb\") pod \"e651d6b1-8f52-457e-b6cd-63517553ac6d\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " Oct 30 19:20:28.339508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339474 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util\") pod \"e651d6b1-8f52-457e-b6cd-63517553ac6d\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " Oct 30 19:20:28.339808 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339517 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle\") pod \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " Oct 30 19:20:28.339808 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339544 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util\") pod \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\" (UID: \"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf\") " Oct 30 19:20:28.339808 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.339586 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle\") pod \"e651d6b1-8f52-457e-b6cd-63517553ac6d\" (UID: \"e651d6b1-8f52-457e-b6cd-63517553ac6d\") " Oct 30 19:20:28.340180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.340128 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle" pod "e651d6b1-8f52-457e-b6cd-63517553ac6d" (UID: "e651d6b1-8f52-457e-b6cd-63517553ac6d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:28.340652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.340626 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle" pod "f100c619-9472-45d5-8e54-146a29da5d19" (UID: "f100c619-9472-45d5-8e54-146a29da5d19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:28.341557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.341529 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle" pod "b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" (UID: "b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:28.341816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.341795 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv" pod "b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" (UID: "b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf"). InnerVolumeSpecName "kube-api-access-b5lzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:28.341992 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.341961 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb" pod "e651d6b1-8f52-457e-b6cd-63517553ac6d" (UID: "e651d6b1-8f52-457e-b6cd-63517553ac6d"). InnerVolumeSpecName "kube-api-access-4cdpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:28.342056 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.342010 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw" pod "f100c619-9472-45d5-8e54-146a29da5d19" (UID: "f100c619-9472-45d5-8e54-146a29da5d19"). InnerVolumeSpecName "kube-api-access-drckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:28.344631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.344611 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util" pod "f100c619-9472-45d5-8e54-146a29da5d19" (UID: "f100c619-9472-45d5-8e54-146a29da5d19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:28.345508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.345487 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util" pod "b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" (UID: "b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:28.345824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.345801 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util" pod "e651d6b1-8f52-457e-b6cd-63517553ac6d" (UID: "e651d6b1-8f52-457e-b6cd-63517553ac6d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:28.440505 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440476 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lzv\" (UniqueName: \"kubernetes.io/projected/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-kube-api-access-b5lzv\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440505 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440499 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdpb\" (UniqueName: \"kubernetes.io/projected/e651d6b1-8f52-457e-b6cd-63517553ac6d-kube-api-access-4cdpb\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440509 6511 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-util\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440519 6511 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440528 6511 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf-util\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440537 6511 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e651d6b1-8f52-457e-b6cd-63517553ac6d-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440545 6511 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-bundle\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440552 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drckw\" (UniqueName: \"kubernetes.io/projected/f100c619-9472-45d5-8e54-146a29da5d19-kube-api-access-drckw\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:28.440646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:28.440560 6511 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f100c619-9472-45d5-8e54-146a29da5d19-util\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:29.015209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.015169 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" event={"ID":"e651d6b1-8f52-457e-b6cd-63517553ac6d","Type":"ContainerDied","Data":"a3208207ed9be92899ed2d162604052c87efa47cf5f46d7928221a8b7815c8d9"} Oct 30 19:20:29.015622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.015249 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3208207ed9be92899ed2d162604052c87efa47cf5f46d7928221a8b7815c8d9" Oct 30 19:20:29.015622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.015184 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448dlff" Oct 30 19:20:29.016882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.016854 6511 generic.go:356] "Generic (PLEG): container finished" podID="46143cf4-b608-4435-9588-d265ebe5f5ab" containerID="b2283bac81cb9973c7377751d1d625e09cab1de7a8ef2ed060bd9dde854c0f87" exitCode=0 Oct 30 19:20:29.017004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.016931 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" event={"ID":"46143cf4-b608-4435-9588-d265ebe5f5ab","Type":"ContainerDied","Data":"b2283bac81cb9973c7377751d1d625e09cab1de7a8ef2ed060bd9dde854c0f87"} Oct 30 19:20:29.019053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.019033 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" event={"ID":"f100c619-9472-45d5-8e54-146a29da5d19","Type":"ContainerDied","Data":"22e463a239db36e59d098d6d94809601f3da3544ea06f1ec784c80b6e0f6a46e"} Oct 30 19:20:29.019123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.019060 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72kmw7" Oct 30 19:20:29.019123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.019069 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22e463a239db36e59d098d6d94809601f3da3544ea06f1ec784c80b6e0f6a46e" Oct 30 19:20:29.020875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.020857 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" event={"ID":"b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf","Type":"ContainerDied","Data":"eda063b8da7c2c4f52e1cc443cbc11f44a7b0fb368f47a10137880926072f0fc"} Oct 30 19:20:29.020966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.020888 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda063b8da7c2c4f52e1cc443cbc11f44a7b0fb368f47a10137880926072f0fc" Oct 30 19:20:29.020966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:29.020908 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azhhgg" Oct 30 19:20:30.026499 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:30.026463 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" event={"ID":"46143cf4-b608-4435-9588-d265ebe5f5ab","Type":"ContainerStarted","Data":"3d8a1c307127886c89f62d60c1124da1afd4de07bbc88f8d012e855adeb61c4a"} Oct 30 19:20:30.026907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:30.026560 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:30.072337 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:30.072270 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" podStartSLOduration=3.088428726 podStartE2EDuration="9.072252434s" totalImagesPullingTime="5.450672778s" totalInitContainerRuntime="533.15093ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:30.071363075 +0000 UTC m=+204.094966973" watchObservedRunningTime="2026-10-30 19:20:30.072252434 +0000 UTC m=+204.095856335" Oct 30 19:20:34.947359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947317 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv"] Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947598 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="util" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947608 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="util" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947619 6511 cpu_manager.go:414] "containerMap: removing container" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947624 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947632 6511 cpu_manager.go:414] "containerMap: removing container" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947637 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947642 6511 cpu_manager.go:414] "containerMap: removing container" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947672 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947681 6511 cpu_manager.go:414] "containerMap: removing container" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="util" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947685 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="util" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947690 6511 cpu_manager.go:414] "containerMap: removing container" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947695 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947701 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947706 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947715 6511 cpu_manager.go:414] "containerMap: removing container" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947719 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="extract" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947728 6511 cpu_manager.go:414] "containerMap: removing container" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947732 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947738 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947742 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947749 6511 cpu_manager.go:414] "containerMap: removing container" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="util" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947753 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="util" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947759 6511 cpu_manager.go:414] "containerMap: removing container" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="pull" Oct 30 19:20:34.947754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947765 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="pull" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947772 6511 cpu_manager.go:414] "containerMap: removing container" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947776 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947784 6511 cpu_manager.go:414] "containerMap: removing container" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="util" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947788 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="util" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947795 6511 cpu_manager.go:414] "containerMap: removing container" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="pull" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947799 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="pull" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947805 6511 cpu_manager.go:414] "containerMap: removing container" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="util" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947809 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="util" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947814 6511 cpu_manager.go:414] "containerMap: removing container" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="util" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947818 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="util" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947827 6511 cpu_manager.go:414] "containerMap: removing container" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947830 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947871 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e07d528-36c8-42bf-b8b7-bcb7196dbb9d" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947880 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a4e301-db40-4d07-9685-0a6a9eeb92b7" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947887 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1be0d18-cd7e-4d9e-a79b-0b7d4f8e1dcf" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947900 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="e651d6b1-8f52-457e-b6cd-63517553ac6d" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947905 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="f100c619-9472-45d5-8e54-146a29da5d19" containerName="extract" Oct 30 19:20:34.948445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.947911 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eeeed4-ee5d-46ef-8792-b0a3f401abbc" containerName="extract" Oct 30 19:20:34.953849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.953831 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:34.961093 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.961076 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:34.961198 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.961177 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kvpwc\"" Oct 30 19:20:34.966485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.966466 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:34.968829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:34.968806 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv"] Oct 30 19:20:35.096471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.096436 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkjq\" (UniqueName: \"kubernetes.io/projected/06bcff7d-30ea-4c30-a99a-ef76420de0fd-kube-api-access-jjkjq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv\" (UID: \"06bcff7d-30ea-4c30-a99a-ef76420de0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.096655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.096492 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06bcff7d-30ea-4c30-a99a-ef76420de0fd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv\" (UID: \"06bcff7d-30ea-4c30-a99a-ef76420de0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.196970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.196934 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkjq\" (UniqueName: \"kubernetes.io/projected/06bcff7d-30ea-4c30-a99a-ef76420de0fd-kube-api-access-jjkjq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv\" (UID: \"06bcff7d-30ea-4c30-a99a-ef76420de0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.197166 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.196980 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06bcff7d-30ea-4c30-a99a-ef76420de0fd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv\" (UID: \"06bcff7d-30ea-4c30-a99a-ef76420de0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.197541 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.197472 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06bcff7d-30ea-4c30-a99a-ef76420de0fd-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv\" (UID: \"06bcff7d-30ea-4c30-a99a-ef76420de0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.225782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.225752 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkjq\" (UniqueName: \"kubernetes.io/projected/06bcff7d-30ea-4c30-a99a-ef76420de0fd-kube-api-access-jjkjq\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv\" (UID: \"06bcff7d-30ea-4c30-a99a-ef76420de0fd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.265242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.265204 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" Oct 30 19:20:35.554099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:35.554069 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv"] Oct 30 19:20:35.554808 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:20:35.554778 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bcff7d_30ea_4c30_a99a_ef76420de0fd.slice/crio-cf800622bc47107be57dfd4d7574a2eb43aa5a15cc0d07222f9725af3b280159 WatchSource:0}: Error finding container cf800622bc47107be57dfd4d7574a2eb43aa5a15cc0d07222f9725af3b280159: Status 404 returned error can't find the container with id cf800622bc47107be57dfd4d7574a2eb43aa5a15cc0d07222f9725af3b280159 Oct 30 19:20:36.055963 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:36.055928 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" event={"ID":"06bcff7d-30ea-4c30-a99a-ef76420de0fd","Type":"ContainerStarted","Data":"cf800622bc47107be57dfd4d7574a2eb43aa5a15cc0d07222f9725af3b280159"} Oct 30 19:20:38.073322 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:38.073289 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" event={"ID":"06bcff7d-30ea-4c30-a99a-ef76420de0fd","Type":"ContainerStarted","Data":"dad0baf97dbfa543ddf7e9761ac0dd914b358dff24be76977890490acf4f399b"} Oct 30 19:20:38.099130 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:38.099080 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-5q8vv" podStartSLOduration=2.104940343 podStartE2EDuration="4.099067652s" totalImagesPullingTime="1.994127309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:38.098324892 +0000 UTC m=+212.121928790" watchObservedRunningTime="2026-10-30 19:20:38.099067652 +0000 UTC m=+212.122671547" Oct 30 19:20:39.585642 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.585607 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl"] Oct 30 19:20:39.589102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.589080 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.608543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.608518 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-tpq2r\"" Oct 30 19:20:39.608745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.608728 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 19:20:39.608791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.608772 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:20:39.670213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.670184 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl"] Oct 30 19:20:39.733650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.733623 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgxp\" (UniqueName: \"kubernetes.io/projected/8e033e69-7a40-4fbe-b894-59f3e9d13a70-kube-api-access-zvgxp\") pod \"jobset-operator-674cd59dc5-rgtfl\" (UID: \"8e033e69-7a40-4fbe-b894-59f3e9d13a70\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.733795 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.733677 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e033e69-7a40-4fbe-b894-59f3e9d13a70-tmp\") pod \"jobset-operator-674cd59dc5-rgtfl\" (UID: \"8e033e69-7a40-4fbe-b894-59f3e9d13a70\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.834068 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.834034 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e033e69-7a40-4fbe-b894-59f3e9d13a70-tmp\") pod \"jobset-operator-674cd59dc5-rgtfl\" (UID: \"8e033e69-7a40-4fbe-b894-59f3e9d13a70\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.834219 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.834196 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgxp\" (UniqueName: \"kubernetes.io/projected/8e033e69-7a40-4fbe-b894-59f3e9d13a70-kube-api-access-zvgxp\") pod \"jobset-operator-674cd59dc5-rgtfl\" (UID: \"8e033e69-7a40-4fbe-b894-59f3e9d13a70\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.834443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.834398 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e033e69-7a40-4fbe-b894-59f3e9d13a70-tmp\") pod \"jobset-operator-674cd59dc5-rgtfl\" (UID: \"8e033e69-7a40-4fbe-b894-59f3e9d13a70\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.859891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.859833 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgxp\" (UniqueName: \"kubernetes.io/projected/8e033e69-7a40-4fbe-b894-59f3e9d13a70-kube-api-access-zvgxp\") pod \"jobset-operator-674cd59dc5-rgtfl\" (UID: \"8e033e69-7a40-4fbe-b894-59f3e9d13a70\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:39.908682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:39.908646 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" Oct 30 19:20:40.188358 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:40.188335 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl"] Oct 30 19:20:40.190098 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:20:40.190063 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e033e69_7a40_4fbe_b894_59f3e9d13a70.slice/crio-d193be716b5a4f040d745f7d149b3d0393b0691bd76416289ffcc90a439d41a8 WatchSource:0}: Error finding container d193be716b5a4f040d745f7d149b3d0393b0691bd76416289ffcc90a439d41a8: Status 404 returned error can't find the container with id d193be716b5a4f040d745f7d149b3d0393b0691bd76416289ffcc90a439d41a8 Oct 30 19:20:41.032513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:41.032483 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-68fb5c5984-qhdzc" Oct 30 19:20:41.086996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:41.086960 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" event={"ID":"8e033e69-7a40-4fbe-b894-59f3e9d13a70","Type":"ContainerStarted","Data":"d193be716b5a4f040d745f7d149b3d0393b0691bd76416289ffcc90a439d41a8"} Oct 30 19:20:43.099101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.099068 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" event={"ID":"8e033e69-7a40-4fbe-b894-59f3e9d13a70","Type":"ContainerStarted","Data":"b38f3b8f54a980da8480098135520ac6a0b4bbe8adcb0318bb51aed622d3b896"} Oct 30 19:20:43.128316 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.128289 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f"] Oct 30 19:20:43.131551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.131507 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rgtfl" podStartSLOduration=2.133942056 podStartE2EDuration="4.131495214s" totalImagesPullingTime="1.997553158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:43.129745509 +0000 UTC m=+217.153349407" watchObservedRunningTime="2026-10-30 19:20:43.131495214 +0000 UTC m=+217.155099112" Oct 30 19:20:43.131756 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.131740 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.149162 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.149142 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mmjsh\"" Oct 30 19:20:43.151660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.151641 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:20:43.151759 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.151698 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:20:43.152147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.152128 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f"] Oct 30 19:20:43.268001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.264152 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51c94d31-00c2-4600-9853-23510cd95eac-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lkj4f\" (UID: \"51c94d31-00c2-4600-9853-23510cd95eac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.268001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.264234 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrk4\" (UniqueName: \"kubernetes.io/projected/51c94d31-00c2-4600-9853-23510cd95eac-kube-api-access-pgrk4\") pod \"cert-manager-cainjector-5c57bbd4fc-lkj4f\" (UID: \"51c94d31-00c2-4600-9853-23510cd95eac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.365402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.365331 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51c94d31-00c2-4600-9853-23510cd95eac-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lkj4f\" (UID: \"51c94d31-00c2-4600-9853-23510cd95eac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.365402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.365376 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrk4\" (UniqueName: \"kubernetes.io/projected/51c94d31-00c2-4600-9853-23510cd95eac-kube-api-access-pgrk4\") pod \"cert-manager-cainjector-5c57bbd4fc-lkj4f\" (UID: \"51c94d31-00c2-4600-9853-23510cd95eac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.390495 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.390465 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrk4\" (UniqueName: \"kubernetes.io/projected/51c94d31-00c2-4600-9853-23510cd95eac-kube-api-access-pgrk4\") pod \"cert-manager-cainjector-5c57bbd4fc-lkj4f\" (UID: \"51c94d31-00c2-4600-9853-23510cd95eac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.393821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.393796 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51c94d31-00c2-4600-9853-23510cd95eac-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lkj4f\" (UID: \"51c94d31-00c2-4600-9853-23510cd95eac\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.442743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.442718 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" Oct 30 19:20:43.739879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:43.739798 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f"] Oct 30 19:20:43.742690 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:20:43.742663 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c94d31_00c2_4600_9853_23510cd95eac.slice/crio-04bdef64e69e469e4ac77923f3d1110ddd69f957004f35f1451ab5ec76e2487b WatchSource:0}: Error finding container 04bdef64e69e469e4ac77923f3d1110ddd69f957004f35f1451ab5ec76e2487b: Status 404 returned error can't find the container with id 04bdef64e69e469e4ac77923f3d1110ddd69f957004f35f1451ab5ec76e2487b Oct 30 19:20:44.103556 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:44.103523 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" event={"ID":"51c94d31-00c2-4600-9853-23510cd95eac","Type":"ContainerStarted","Data":"04bdef64e69e469e4ac77923f3d1110ddd69f957004f35f1451ab5ec76e2487b"} Oct 30 19:20:47.118982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.118943 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" event={"ID":"51c94d31-00c2-4600-9853-23510cd95eac","Type":"ContainerStarted","Data":"b0793ec7cc40830f73f9120f29fc6a2af8ff8cdb6565c802bb27fef8f70dbe49"} Oct 30 19:20:47.141181 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.141117 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lkj4f" podStartSLOduration=1.352458984 podStartE2EDuration="4.141100826s" totalImagesPullingTime="2.788641842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:47.140099824 +0000 UTC m=+221.163703722" watchObservedRunningTime="2026-10-30 19:20:47.141100826 +0000 UTC m=+221.164704746" Oct 30 19:20:47.691592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.691549 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s"] Oct 30 19:20:47.695002 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.694985 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:47.700268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.700245 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gmcx4\"" Oct 30 19:20:47.707870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.707849 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s"] Oct 30 19:20:47.802923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.802895 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd512f66-1e42-4a8d-bd20-0a461fefe100-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z6m7s\" (UID: \"cd512f66-1e42-4a8d-bd20-0a461fefe100\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:47.802923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.802923 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6g4\" (UniqueName: \"kubernetes.io/projected/cd512f66-1e42-4a8d-bd20-0a461fefe100-kube-api-access-bp6g4\") pod \"cert-manager-webhook-5dbbbf75cc-z6m7s\" (UID: \"cd512f66-1e42-4a8d-bd20-0a461fefe100\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:47.903375 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.903336 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd512f66-1e42-4a8d-bd20-0a461fefe100-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z6m7s\" (UID: \"cd512f66-1e42-4a8d-bd20-0a461fefe100\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:47.903375 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.903374 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6g4\" (UniqueName: \"kubernetes.io/projected/cd512f66-1e42-4a8d-bd20-0a461fefe100-kube-api-access-bp6g4\") pod \"cert-manager-webhook-5dbbbf75cc-z6m7s\" (UID: \"cd512f66-1e42-4a8d-bd20-0a461fefe100\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:47.913232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.913200 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6g4\" (UniqueName: \"kubernetes.io/projected/cd512f66-1e42-4a8d-bd20-0a461fefe100-kube-api-access-bp6g4\") pod \"cert-manager-webhook-5dbbbf75cc-z6m7s\" (UID: \"cd512f66-1e42-4a8d-bd20-0a461fefe100\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:47.913326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:47.913273 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd512f66-1e42-4a8d-bd20-0a461fefe100-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-z6m7s\" (UID: \"cd512f66-1e42-4a8d-bd20-0a461fefe100\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:48.006990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:48.006909 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:48.308119 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:48.308093 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s"] Oct 30 19:20:48.309794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:20:48.309767 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd512f66_1e42_4a8d_bd20_0a461fefe100.slice/crio-7f8ad12f401fa0e53b574b9bd16a831175274e155586f34acf3cf03ba1e42d3a WatchSource:0}: Error finding container 7f8ad12f401fa0e53b574b9bd16a831175274e155586f34acf3cf03ba1e42d3a: Status 404 returned error can't find the container with id 7f8ad12f401fa0e53b574b9bd16a831175274e155586f34acf3cf03ba1e42d3a Oct 30 19:20:49.128739 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.128707 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" event={"ID":"cd512f66-1e42-4a8d-bd20-0a461fefe100","Type":"ContainerStarted","Data":"8f097c80ac7c18cdb977ead83e179804f516f5bd5598fa71fc306863fa0e5869"} Oct 30 19:20:49.128881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.128746 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" event={"ID":"cd512f66-1e42-4a8d-bd20-0a461fefe100","Type":"ContainerStarted","Data":"7f8ad12f401fa0e53b574b9bd16a831175274e155586f34acf3cf03ba1e42d3a"} Oct 30 19:20:49.128881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.128787 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:20:49.163472 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.163405 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" podStartSLOduration=2.16339201 podStartE2EDuration="2.16339201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:49.163093739 +0000 UTC m=+223.186697638" watchObservedRunningTime="2026-10-30 19:20:49.16339201 +0000 UTC m=+223.186995908" Oct 30 19:20:49.929187 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.929155 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-lnnpc"] Oct 30 19:20:49.933516 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.933500 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:49.945653 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.945609 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-lnnpc"] Oct 30 19:20:49.945780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:49.945766 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7lndm\"" Oct 30 19:20:50.021446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.021396 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rfn\" (UniqueName: \"kubernetes.io/projected/064e4b62-13ca-4c3a-b72b-b21dd6d1af02-kube-api-access-v5rfn\") pod \"cert-manager-597b95cb84-lnnpc\" (UID: \"064e4b62-13ca-4c3a-b72b-b21dd6d1af02\") " pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.021625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.021532 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/064e4b62-13ca-4c3a-b72b-b21dd6d1af02-bound-sa-token\") pod \"cert-manager-597b95cb84-lnnpc\" (UID: \"064e4b62-13ca-4c3a-b72b-b21dd6d1af02\") " pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.122790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.122758 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rfn\" (UniqueName: \"kubernetes.io/projected/064e4b62-13ca-4c3a-b72b-b21dd6d1af02-kube-api-access-v5rfn\") pod \"cert-manager-597b95cb84-lnnpc\" (UID: \"064e4b62-13ca-4c3a-b72b-b21dd6d1af02\") " pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.122924 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.122822 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/064e4b62-13ca-4c3a-b72b-b21dd6d1af02-bound-sa-token\") pod \"cert-manager-597b95cb84-lnnpc\" (UID: \"064e4b62-13ca-4c3a-b72b-b21dd6d1af02\") " pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.135019 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.134991 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/064e4b62-13ca-4c3a-b72b-b21dd6d1af02-bound-sa-token\") pod \"cert-manager-597b95cb84-lnnpc\" (UID: \"064e4b62-13ca-4c3a-b72b-b21dd6d1af02\") " pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.135334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.135313 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rfn\" (UniqueName: \"kubernetes.io/projected/064e4b62-13ca-4c3a-b72b-b21dd6d1af02-kube-api-access-v5rfn\") pod \"cert-manager-597b95cb84-lnnpc\" (UID: \"064e4b62-13ca-4c3a-b72b-b21dd6d1af02\") " pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.246512 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.246435 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-lnnpc" Oct 30 19:20:50.516922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:50.516899 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-lnnpc"] Oct 30 19:20:51.138758 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:51.138722 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-lnnpc" event={"ID":"064e4b62-13ca-4c3a-b72b-b21dd6d1af02","Type":"ContainerStarted","Data":"074f07250a4480fd53542369bcf2669cb59e93422b1fe770fc9f19747a172d02"} Oct 30 19:20:51.138758 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:51.138764 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-lnnpc" event={"ID":"064e4b62-13ca-4c3a-b72b-b21dd6d1af02","Type":"ContainerStarted","Data":"2bb9db5e25d59dfa5ee9c514a90a0f2bc314ed0ab86e52257833e41d86921f8e"} Oct 30 19:20:51.203364 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:51.203315 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-lnnpc" podStartSLOduration=2.203299439 podStartE2EDuration="2.203299439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:51.201037021 +0000 UTC m=+225.224640917" watchObservedRunningTime="2026-10-30 19:20:51.203299439 +0000 UTC m=+225.226903341" Oct 30 19:20:54.375889 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:20:54.375852 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:20:55.135978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:20:55.135948 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" Oct 30 19:21:06.546741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.546707 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s"] Oct 30 19:21:06.552790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.552769 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.631882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.631857 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:21:06.632111 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.632089 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qtqt2\"" Oct 30 19:21:06.632277 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.632261 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:21:06.655827 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.655796 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-tmp\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.655991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.655879 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dcx\" (UniqueName: \"kubernetes.io/projected/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-kube-api-access-d2dcx\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.655991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.655972 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-tls-certs\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.659531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.658949 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s"] Oct 30 19:21:06.757289 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.757250 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dcx\" (UniqueName: \"kubernetes.io/projected/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-kube-api-access-d2dcx\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.757488 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.757301 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-tls-certs\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.757488 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.757361 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-tmp\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.759831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.759799 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-tmp\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.760288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.760049 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-tls-certs\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.769007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.768986 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dcx\" (UniqueName: \"kubernetes.io/projected/82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43-kube-api-access-d2dcx\") pod \"kube-auth-proxy-7c5f4bd96d-fsk2s\" (UID: \"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43\") " pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:06.863922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:06.863877 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" Oct 30 19:21:07.147827 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.147801 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s"] Oct 30 19:21:07.149936 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:21:07.149888 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bbe7d5_5b5b_41cc_bb54_2b144b9d7b43.slice/crio-efe6f2f08bda014c18786e160634b63406a4f1616e0d3e07eb8bc72312508809 WatchSource:0}: Error finding container efe6f2f08bda014c18786e160634b63406a4f1616e0d3e07eb8bc72312508809: Status 404 returned error can't find the container with id efe6f2f08bda014c18786e160634b63406a4f1616e0d3e07eb8bc72312508809 Oct 30 19:21:07.209546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.209515 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" event={"ID":"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43","Type":"ContainerStarted","Data":"efe6f2f08bda014c18786e160634b63406a4f1616e0d3e07eb8bc72312508809"} Oct 30 19:21:07.583583 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.583549 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:07.586131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.586113 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.597409 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.597382 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:21:07.598577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.598549 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:21:07.598705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.598656 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:21:07.600005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.599982 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:07.650782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.650755 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wz84h\"" Oct 30 19:21:07.664457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.664427 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file\") pod \"limitador-limitador-74dfcfb894-wzrfx\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.664604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.664488 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5s2k\" (UniqueName: \"kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k\") pod \"limitador-limitador-74dfcfb894-wzrfx\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.681354 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.681320 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:07.766070 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.766034 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file\") pod \"limitador-limitador-74dfcfb894-wzrfx\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.766241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.766120 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5s2k\" (UniqueName: \"kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k\") pod \"limitador-limitador-74dfcfb894-wzrfx\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.766767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.766743 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file\") pod \"limitador-limitador-74dfcfb894-wzrfx\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.778572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.778541 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5s2k\" (UniqueName: \"kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k\") pod \"limitador-limitador-74dfcfb894-wzrfx\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:07.898108 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:07.898016 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:08.180377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:08.180308 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:08.184211 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:21:08.184181 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7852d073_eda1_45b6_8bb3_09bee59b43bb.slice/crio-dcf47a1f83818152580cd444b33ff643c25402f96e42e3a4987cf96d38bd20bb WatchSource:0}: Error finding container dcf47a1f83818152580cd444b33ff643c25402f96e42e3a4987cf96d38bd20bb: Status 404 returned error can't find the container with id dcf47a1f83818152580cd444b33ff643c25402f96e42e3a4987cf96d38bd20bb Oct 30 19:21:08.215589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:08.215556 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" event={"ID":"7852d073-eda1-45b6-8bb3-09bee59b43bb","Type":"ContainerStarted","Data":"dcf47a1f83818152580cd444b33ff643c25402f96e42e3a4987cf96d38bd20bb"} Oct 30 19:21:10.228154 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.228118 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" event={"ID":"82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43","Type":"ContainerStarted","Data":"cab38480c7f030b78a7a81e30b01860b877a67008789017dd400c28759b4ba90"} Oct 30 19:21:10.252394 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.252347 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c5f4bd96d-fsk2s" podStartSLOduration=1.7302420920000001 podStartE2EDuration="4.252332534s" totalImagesPullingTime="2.522090442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:10.250989282 +0000 UTC m=+244.274593186" watchObservedRunningTime="2026-10-30 19:21:10.252332534 +0000 UTC m=+244.275936432" Oct 30 19:21:10.869063 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.869036 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm"] Oct 30 19:21:10.872182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.872166 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:10.877861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.877841 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:21:10.877958 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.877842 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:21:10.877958 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.877863 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gnp84\"" Oct 30 19:21:10.883831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.883814 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 19:21:10.886886 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.886868 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm"] Oct 30 19:21:10.998259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.998175 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03278d76-60bf-42ad-acd7-0ad099d16707-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:10.998259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.998241 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03278d76-60bf-42ad-acd7-0ad099d16707-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:10.998486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.998330 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5r5\" (UniqueName: \"kubernetes.io/projected/03278d76-60bf-42ad-acd7-0ad099d16707-kube-api-access-jw5r5\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:10.998486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:10.998375 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03278d76-60bf-42ad-acd7-0ad099d16707-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.099494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.099466 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03278d76-60bf-42ad-acd7-0ad099d16707-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.099494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.099507 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5r5\" (UniqueName: \"kubernetes.io/projected/03278d76-60bf-42ad-acd7-0ad099d16707-kube-api-access-jw5r5\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.099702 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.099538 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03278d76-60bf-42ad-acd7-0ad099d16707-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.099702 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.099576 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03278d76-60bf-42ad-acd7-0ad099d16707-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.099877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.099858 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03278d76-60bf-42ad-acd7-0ad099d16707-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.099994 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.099972 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03278d76-60bf-42ad-acd7-0ad099d16707-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.100467 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.100449 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03278d76-60bf-42ad-acd7-0ad099d16707-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.109745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.109723 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5r5\" (UniqueName: \"kubernetes.io/projected/03278d76-60bf-42ad-acd7-0ad099d16707-kube-api-access-jw5r5\") pod \"kserve-module-controller-manager-7566cf5fc8-nhvrm\" (UID: \"03278d76-60bf-42ad-acd7-0ad099d16707\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.184645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.184620 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:11.246500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.246451 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" event={"ID":"7852d073-eda1-45b6-8bb3-09bee59b43bb","Type":"ContainerStarted","Data":"f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf"} Oct 30 19:21:11.246893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.246697 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:11.266489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.266381 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" podStartSLOduration=1.705013438 podStartE2EDuration="4.266367852s" totalImagesPullingTime="2.561354414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:11.265494636 +0000 UTC m=+245.289098538" watchObservedRunningTime="2026-10-30 19:21:11.266367852 +0000 UTC m=+245.289971751" Oct 30 19:21:11.469368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:11.469344 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm"] Oct 30 19:21:11.471794 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:21:11.471768 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03278d76_60bf_42ad_acd7_0ad099d16707.slice/crio-1ae5c67773813a7f006df9a84c5c8b8a4e4fc2d50af181c0bd776b0dde79586a WatchSource:0}: Error finding container 1ae5c67773813a7f006df9a84c5c8b8a4e4fc2d50af181c0bd776b0dde79586a: Status 404 returned error can't find the container with id 1ae5c67773813a7f006df9a84c5c8b8a4e4fc2d50af181c0bd776b0dde79586a Oct 30 19:21:12.253712 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:12.253666 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" event={"ID":"03278d76-60bf-42ad-acd7-0ad099d16707","Type":"ContainerStarted","Data":"1ae5c67773813a7f006df9a84c5c8b8a4e4fc2d50af181c0bd776b0dde79586a"} Oct 30 19:21:13.260019 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:13.259942 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" event={"ID":"03278d76-60bf-42ad-acd7-0ad099d16707","Type":"ContainerStarted","Data":"4c7438954597d68cc2ef8006c060a8f45565ee58cbe374cd90eca4698a846f42"} Oct 30 19:21:13.260478 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:13.260041 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:13.314137 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:13.314090 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" podStartSLOduration=1.904241278 podStartE2EDuration="3.314077943s" totalImagesPullingTime="1.409836665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:13.312314418 +0000 UTC m=+247.335918317" watchObservedRunningTime="2026-10-30 19:21:13.314077943 +0000 UTC m=+247.337681841" Oct 30 19:21:22.255092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:22.255063 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:24.189835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.189793 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:24.190386 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.190016 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" podUID="7852d073-eda1-45b6-8bb3-09bee59b43bb" containerName="limitador" containerID="cri-o://f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf" gracePeriod=30 Oct 30 19:21:24.376392 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:21:24.376342 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:21:24.735738 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.735715 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:24.912975 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.912943 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k\" (UniqueName: \"kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k\") pod \"7852d073-eda1-45b6-8bb3-09bee59b43bb\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " Oct 30 19:21:24.913175 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.913057 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file\" (UniqueName: \"kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file\") pod \"7852d073-eda1-45b6-8bb3-09bee59b43bb\" (UID: \"7852d073-eda1-45b6-8bb3-09bee59b43bb\") " Oct 30 19:21:24.913489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.913459 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file" pod "7852d073-eda1-45b6-8bb3-09bee59b43bb" (UID: "7852d073-eda1-45b6-8bb3-09bee59b43bb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:24.915473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:24.915441 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k" pod "7852d073-eda1-45b6-8bb3-09bee59b43bb" (UID: "7852d073-eda1-45b6-8bb3-09bee59b43bb"). InnerVolumeSpecName "kube-api-access-w5s2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:25.014025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.013991 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5s2k\" (UniqueName: \"kubernetes.io/projected/7852d073-eda1-45b6-8bb3-09bee59b43bb-kube-api-access-w5s2k\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:25.014025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.014024 6511 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7852d073-eda1-45b6-8bb3-09bee59b43bb-config-file\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:25.314558 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.314523 6511 generic.go:356] "Generic (PLEG): container finished" podID="7852d073-eda1-45b6-8bb3-09bee59b43bb" containerID="f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf" exitCode=0 Oct 30 19:21:25.315033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.314582 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" Oct 30 19:21:25.315033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.314587 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" event={"ID":"7852d073-eda1-45b6-8bb3-09bee59b43bb","Type":"ContainerDied","Data":"f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf"} Oct 30 19:21:25.315033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.314712 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wzrfx" event={"ID":"7852d073-eda1-45b6-8bb3-09bee59b43bb","Type":"ContainerDied","Data":"dcf47a1f83818152580cd444b33ff643c25402f96e42e3a4987cf96d38bd20bb"} Oct 30 19:21:25.315033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.314743 6511 scope.go:122] "RemoveContainer" containerID="f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf" Oct 30 19:21:25.323878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.323855 6511 scope.go:122] "RemoveContainer" containerID="f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf" Oct 30 19:21:25.324155 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:21:25.324138 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf\": container with ID starting with f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf not found: ID does not exist" containerID="f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf" Oct 30 19:21:25.324209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.324165 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf"} err="failed to get container status \"f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf\": rpc error: code = NotFound desc = could not find container \"f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf\": container with ID starting with f891b0e1d9f90e4725782c448501eea5c8309a517dfdea518ee5f65a5cc840bf not found: ID does not exist" Oct 30 19:21:25.340919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.340897 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:25.344828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:25.344807 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wzrfx"] Oct 30 19:21:26.095702 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:26.095674 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7852d073-eda1-45b6-8bb3-09bee59b43bb" path="/var/lib/kubelet/pods/7852d073-eda1-45b6-8bb3-09bee59b43bb/volumes" Oct 30 19:21:29.549952 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.549916 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c"] Oct 30 19:21:29.550336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.550269 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7852d073-eda1-45b6-8bb3-09bee59b43bb" containerName="limitador" Oct 30 19:21:29.550336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.550281 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7852d073-eda1-45b6-8bb3-09bee59b43bb" containerName="limitador" Oct 30 19:21:29.550408 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.550347 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7852d073-eda1-45b6-8bb3-09bee59b43bb" containerName="limitador" Oct 30 19:21:29.554652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.554634 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.571587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.571560 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:21:29.571707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.571606 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:21:29.571707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.571629 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 19:21:29.573125 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.573101 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kk7m9\"" Oct 30 19:21:29.574482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.574460 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c"] Oct 30 19:21:29.657434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.657392 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6v2\" (UniqueName: \"kubernetes.io/projected/b4483e14-f12b-4836-afb5-e6ca25af6663-kube-api-access-2s6v2\") pod \"postgres-7cdbf6dc44-k4w5c\" (UID: \"b4483e14-f12b-4836-afb5-e6ca25af6663\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.657596 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.657499 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4483e14-f12b-4836-afb5-e6ca25af6663-data\") pod \"postgres-7cdbf6dc44-k4w5c\" (UID: \"b4483e14-f12b-4836-afb5-e6ca25af6663\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.758959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.758925 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6v2\" (UniqueName: \"kubernetes.io/projected/b4483e14-f12b-4836-afb5-e6ca25af6663-kube-api-access-2s6v2\") pod \"postgres-7cdbf6dc44-k4w5c\" (UID: \"b4483e14-f12b-4836-afb5-e6ca25af6663\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.759103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.758964 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4483e14-f12b-4836-afb5-e6ca25af6663-data\") pod \"postgres-7cdbf6dc44-k4w5c\" (UID: \"b4483e14-f12b-4836-afb5-e6ca25af6663\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.759292 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.759275 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4483e14-f12b-4836-afb5-e6ca25af6663-data\") pod \"postgres-7cdbf6dc44-k4w5c\" (UID: \"b4483e14-f12b-4836-afb5-e6ca25af6663\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.768640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.768614 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6v2\" (UniqueName: \"kubernetes.io/projected/b4483e14-f12b-4836-afb5-e6ca25af6663-kube-api-access-2s6v2\") pod \"postgres-7cdbf6dc44-k4w5c\" (UID: \"b4483e14-f12b-4836-afb5-e6ca25af6663\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:29.866367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:29.866304 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:30.141688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:30.141666 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c"] Oct 30 19:21:30.143632 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:21:30.143605 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4483e14_f12b_4836_afb5_e6ca25af6663.slice/crio-6848306e05b1b638ee2c70f77f11c93b84920de5933d2d6737504f3b4e5c4f8b WatchSource:0}: Error finding container 6848306e05b1b638ee2c70f77f11c93b84920de5933d2d6737504f3b4e5c4f8b: Status 404 returned error can't find the container with id 6848306e05b1b638ee2c70f77f11c93b84920de5933d2d6737504f3b4e5c4f8b Oct 30 19:21:30.345284 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:30.345245 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" event={"ID":"b4483e14-f12b-4836-afb5-e6ca25af6663","Type":"ContainerStarted","Data":"6848306e05b1b638ee2c70f77f11c93b84920de5933d2d6737504f3b4e5c4f8b"} Oct 30 19:21:34.266439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:34.266390 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-nhvrm" Oct 30 19:21:36.378384 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:36.378350 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" event={"ID":"b4483e14-f12b-4836-afb5-e6ca25af6663","Type":"ContainerStarted","Data":"729c8f73d40524199b7dda6ce677ec3f135b852a4d56b91b356796b60883afa1"} Oct 30 19:21:36.378791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:36.378475 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:36.400572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:36.400529 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" podStartSLOduration=1.276839861 podStartE2EDuration="7.400516117s" totalImagesPullingTime="6.123676256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:36.398620938 +0000 UTC m=+270.422224837" watchObservedRunningTime="2026-10-30 19:21:36.400516117 +0000 UTC m=+270.424120058" Oct 30 19:21:42.411713 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:42.411682 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-k4w5c" Oct 30 19:21:49.598853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.598772 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9v76w"] Oct 30 19:21:49.602297 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.602280 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.609914 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.609898 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:21:49.614638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.614611 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:21:49.614638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.614616 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5rr7g\"" Oct 30 19:21:49.615005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.614978 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9v76w"] Oct 30 19:21:49.615111 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.615099 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:21:49.737537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.737501 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.737537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.737542 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.737782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.737578 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrfx\" (UniqueName: \"kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.838272 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.838233 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.838436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.838276 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.838436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.838310 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrfx\" (UniqueName: \"kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.840858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.840840 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.841080 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.841056 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.848010 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.847985 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrfx\" (UniqueName: \"kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx\") pod \"maas-controller-579cf7c6d-9v76w\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:49.916058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:49.915987 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:50.196839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:50.196762 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9v76w"] Oct 30 19:21:50.200652 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:21:50.200619 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3733b4db_9619_48fa_aae1_9c37f03eab00.slice/crio-c75a1f9bcf0542b890d65c917d8156c7d9f0d29cf50dc5bd6b3018ada4290083 WatchSource:0}: Error finding container c75a1f9bcf0542b890d65c917d8156c7d9f0d29cf50dc5bd6b3018ada4290083: Status 404 returned error can't find the container with id c75a1f9bcf0542b890d65c917d8156c7d9f0d29cf50dc5bd6b3018ada4290083 Oct 30 19:21:50.442577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:50.442538 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9v76w" event={"ID":"3733b4db-9619-48fa-aae1-9c37f03eab00","Type":"ContainerStarted","Data":"c75a1f9bcf0542b890d65c917d8156c7d9f0d29cf50dc5bd6b3018ada4290083"} Oct 30 19:21:53.461003 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:53.460964 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9v76w" event={"ID":"3733b4db-9619-48fa-aae1-9c37f03eab00","Type":"ContainerStarted","Data":"f1e0a1d84bef272ae2f8a9d611919a1014b91fcd2408e11cd4726f08bebcb45a"} Oct 30 19:21:53.461495 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:53.461097 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:21:53.484009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:21:53.483966 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-9v76w" podStartSLOduration=1.447888955 podStartE2EDuration="4.483953755s" totalImagesPullingTime="3.0360648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:53.482067543 +0000 UTC m=+287.505671465" watchObservedRunningTime="2026-10-30 19:21:53.483953755 +0000 UTC m=+287.507557654" Oct 30 19:21:54.378013 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:21:54.377976 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:22:04.373247 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.373207 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9v76w"] Oct 30 19:22:04.373781 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.373429 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-9v76w" podUID="3733b4db-9619-48fa-aae1-9c37f03eab00" containerName="manager" containerID="cri-o://f1e0a1d84bef272ae2f8a9d611919a1014b91fcd2408e11cd4726f08bebcb45a" gracePeriod=30 Oct 30 19:22:04.375369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.375323 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:22:04.453525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.453491 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-79xzr"] Oct 30 19:22:04.459409 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.459391 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.471320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.471299 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gdcnr\"" Oct 30 19:22:04.471320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.471306 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 19:22:04.491071 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.491050 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-79xzr"] Oct 30 19:22:04.516047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.515910 6511 generic.go:356] "Generic (PLEG): container finished" podID="3733b4db-9619-48fa-aae1-9c37f03eab00" containerID="f1e0a1d84bef272ae2f8a9d611919a1014b91fcd2408e11cd4726f08bebcb45a" exitCode=0 Oct 30 19:22:04.516047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.515982 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9v76w" event={"ID":"3733b4db-9619-48fa-aae1-9c37f03eab00","Type":"ContainerDied","Data":"f1e0a1d84bef272ae2f8a9d611919a1014b91fcd2408e11cd4726f08bebcb45a"} Oct 30 19:22:04.553343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.553315 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b"] Oct 30 19:22:04.563900 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.563877 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.570630 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.570599 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzkg\" (UniqueName: \"kubernetes.io/projected/14ed5071-2475-47ad-b73b-83519f55840e-kube-api-access-zpzkg\") pod \"kserve-controller-manager-776cc66d47-79xzr\" (UID: \"14ed5071-2475-47ad-b73b-83519f55840e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.570765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.570724 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14ed5071-2475-47ad-b73b-83519f55840e-cert\") pod \"kserve-controller-manager-776cc66d47-79xzr\" (UID: \"14ed5071-2475-47ad-b73b-83519f55840e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.598445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.591372 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lw8dt\"" Oct 30 19:22:04.598445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.591553 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 19:22:04.598445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.593037 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b"] Oct 30 19:22:04.630898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.630843 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:22:04.653280 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.653252 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-s6mpf"] Oct 30 19:22:04.653660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.653643 6511 cpu_manager.go:414] "containerMap: removing container" podUID="3733b4db-9619-48fa-aae1-9c37f03eab00" containerName="manager" Oct 30 19:22:04.653660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.653659 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3733b4db-9619-48fa-aae1-9c37f03eab00" containerName="manager" Oct 30 19:22:04.653833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.653716 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="3733b4db-9619-48fa-aae1-9c37f03eab00" containerName="manager" Oct 30 19:22:04.658443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.658397 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.671467 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.671439 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzkg\" (UniqueName: \"kubernetes.io/projected/14ed5071-2475-47ad-b73b-83519f55840e-kube-api-access-zpzkg\") pod \"kserve-controller-manager-776cc66d47-79xzr\" (UID: \"14ed5071-2475-47ad-b73b-83519f55840e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.671625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.671501 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14ed5071-2475-47ad-b73b-83519f55840e-cert\") pod \"kserve-controller-manager-776cc66d47-79xzr\" (UID: \"14ed5071-2475-47ad-b73b-83519f55840e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.671625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.671536 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsbd\" (UniqueName: \"kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd\") pod \"llmisvc-controller-manager-869d5dcc5d-xv82b\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.671625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.671580 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xv82b\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.674463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.674442 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14ed5071-2475-47ad-b73b-83519f55840e-cert\") pod \"kserve-controller-manager-776cc66d47-79xzr\" (UID: \"14ed5071-2475-47ad-b73b-83519f55840e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.685822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.685801 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzkg\" (UniqueName: \"kubernetes.io/projected/14ed5071-2475-47ad-b73b-83519f55840e-kube-api-access-zpzkg\") pod \"kserve-controller-manager-776cc66d47-79xzr\" (UID: \"14ed5071-2475-47ad-b73b-83519f55840e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.690949 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.690927 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 19:22:04.691065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.690937 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 19:22:04.691123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.691058 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nqpgf\"" Oct 30 19:22:04.704531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.704509 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s6mpf"] Oct 30 19:22:04.772193 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772165 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs\") pod \"3733b4db-9619-48fa-aae1-9c37f03eab00\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " Oct 30 19:22:04.772324 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772300 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx\" (UniqueName: \"kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx\") pod \"3733b4db-9619-48fa-aae1-9c37f03eab00\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " Oct 30 19:22:04.772383 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772332 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert\") pod \"3733b4db-9619-48fa-aae1-9c37f03eab00\" (UID: \"3733b4db-9619-48fa-aae1-9c37f03eab00\") " Oct 30 19:22:04.772468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772430 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6s5\" (UniqueName: \"kubernetes.io/projected/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-kube-api-access-ck6s5\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.772528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772480 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsbd\" (UniqueName: \"kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd\") pod \"llmisvc-controller-manager-869d5dcc5d-xv82b\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.772528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772511 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-cert\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.772643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772557 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xv82b\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.772812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.772771 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-metrics-certs\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.774743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.774717 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert" pod "3733b4db-9619-48fa-aae1-9c37f03eab00" (UID: "3733b4db-9619-48fa-aae1-9c37f03eab00"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:04.774845 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.774735 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs" pod "3733b4db-9619-48fa-aae1-9c37f03eab00" (UID: "3733b4db-9619-48fa-aae1-9c37f03eab00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:04.774845 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.774735 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx" pod "3733b4db-9619-48fa-aae1-9c37f03eab00" (UID: "3733b4db-9619-48fa-aae1-9c37f03eab00"). InnerVolumeSpecName "kube-api-access-mkrfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:04.775198 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.775182 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xv82b\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.789106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.789083 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:04.800122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.800104 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsbd\" (UniqueName: \"kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd\") pod \"llmisvc-controller-manager-869d5dcc5d-xv82b\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.874073 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.874039 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-metrics-certs\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.874210 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.874133 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6s5\" (UniqueName: \"kubernetes.io/projected/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-kube-api-access-ck6s5\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.874210 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.874170 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-cert\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.874296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.874218 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:04.874379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.874354 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrfx\" (UniqueName: \"kubernetes.io/projected/3733b4db-9619-48fa-aae1-9c37f03eab00-kube-api-access-mkrfx\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:04.874490 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.874461 6511 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3733b4db-9619-48fa-aae1-9c37f03eab00-webhook-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:04.874490 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:22:04.874463 6511 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Oct 30 19:22:04.874593 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:22:04.874575 6511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-cert podName:67077e2a-7385-4c9c-8f36-ec005a1c2b7d nodeName:}" failed. No retries permitted until 2026-10-30 19:22:05.374551836 +0000 UTC m=+299.398155733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-cert") pod "odh-model-controller-769875db56-s6mpf" (UID: "67077e2a-7385-4c9c-8f36-ec005a1c2b7d") : secret "odh-model-controller-webhook-cert" not found Oct 30 19:22:04.876692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.876670 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-metrics-certs\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:04.881550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.881495 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:04.885241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:04.885216 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6s5\" (UniqueName: \"kubernetes.io/projected/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-kube-api-access-ck6s5\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:05.070378 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.070353 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-79xzr"] Oct 30 19:22:05.072271 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:22:05.072240 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ed5071_2475_47ad_b73b_83519f55840e.slice/crio-78efb38377e38b77dad32bef47b5a1654575346c4363f9bb43784c07ac9dfa2a WatchSource:0}: Error finding container 78efb38377e38b77dad32bef47b5a1654575346c4363f9bb43784c07ac9dfa2a: Status 404 returned error can't find the container with id 78efb38377e38b77dad32bef47b5a1654575346c4363f9bb43784c07ac9dfa2a Oct 30 19:22:05.202925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.202897 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b"] Oct 30 19:22:05.203561 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:22:05.203537 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb0b19bb_d8b5_42a1_9854_ba579b87ffd7.slice/crio-297c2195a4460a2c6014f0e48216e821374740ebd3188ec37ee2fed1ecc7a7e1 WatchSource:0}: Error finding container 297c2195a4460a2c6014f0e48216e821374740ebd3188ec37ee2fed1ecc7a7e1: Status 404 returned error can't find the container with id 297c2195a4460a2c6014f0e48216e821374740ebd3188ec37ee2fed1ecc7a7e1 Oct 30 19:22:05.379952 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.379914 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-cert\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:05.382358 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.382330 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67077e2a-7385-4c9c-8f36-ec005a1c2b7d-cert\") pod \"odh-model-controller-769875db56-s6mpf\" (UID: \"67077e2a-7385-4c9c-8f36-ec005a1c2b7d\") " pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:05.522329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.522257 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9v76w" Oct 30 19:22:05.522518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.522249 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9v76w" event={"ID":"3733b4db-9619-48fa-aae1-9c37f03eab00","Type":"ContainerDied","Data":"c75a1f9bcf0542b890d65c917d8156c7d9f0d29cf50dc5bd6b3018ada4290083"} Oct 30 19:22:05.522518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.522392 6511 scope.go:122] "RemoveContainer" containerID="f1e0a1d84bef272ae2f8a9d611919a1014b91fcd2408e11cd4726f08bebcb45a" Oct 30 19:22:05.523898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.523859 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" event={"ID":"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7","Type":"ContainerStarted","Data":"297c2195a4460a2c6014f0e48216e821374740ebd3188ec37ee2fed1ecc7a7e1"} Oct 30 19:22:05.525531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.525507 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" event={"ID":"14ed5071-2475-47ad-b73b-83519f55840e","Type":"ContainerStarted","Data":"78efb38377e38b77dad32bef47b5a1654575346c4363f9bb43784c07ac9dfa2a"} Oct 30 19:22:05.548001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.547979 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9v76w"] Oct 30 19:22:05.553509 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.553478 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9v76w"] Oct 30 19:22:05.572862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.572834 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:05.862324 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:05.862229 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s6mpf"] Oct 30 19:22:05.864899 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:22:05.864871 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67077e2a_7385_4c9c_8f36_ec005a1c2b7d.slice/crio-82f4da7ae9d119badc7ed23338163ee739d6800dae3421ed1b5f5f466956d0bd WatchSource:0}: Error finding container 82f4da7ae9d119badc7ed23338163ee739d6800dae3421ed1b5f5f466956d0bd: Status 404 returned error can't find the container with id 82f4da7ae9d119badc7ed23338163ee739d6800dae3421ed1b5f5f466956d0bd Oct 30 19:22:06.099042 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:06.099005 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3733b4db-9619-48fa-aae1-9c37f03eab00" path="/var/lib/kubelet/pods/3733b4db-9619-48fa-aae1-9c37f03eab00/volumes" Oct 30 19:22:06.130806 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:06.130781 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:22:06.132613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:06.132592 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:22:06.535458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:06.535351 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s6mpf" event={"ID":"67077e2a-7385-4c9c-8f36-ec005a1c2b7d","Type":"ContainerStarted","Data":"82f4da7ae9d119badc7ed23338163ee739d6800dae3421ed1b5f5f466956d0bd"} Oct 30 19:22:07.543601 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:07.543532 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" event={"ID":"14ed5071-2475-47ad-b73b-83519f55840e","Type":"ContainerStarted","Data":"af22b0e4eb70a0d7d0a5f63aeddc3dc26a3dec8af4291882bd24c29ab5aef1cc"} Oct 30 19:22:07.544122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:07.544086 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:07.583378 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:07.583318 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" podStartSLOduration=1.256564967 podStartE2EDuration="3.583304495s" totalImagesPullingTime="2.326739528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:07.582609041 +0000 UTC m=+301.606212941" watchObservedRunningTime="2026-10-30 19:22:07.583304495 +0000 UTC m=+301.606908394" Oct 30 19:22:08.554954 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:08.554913 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" event={"ID":"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7","Type":"ContainerStarted","Data":"a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5"} Oct 30 19:22:08.555357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:08.555253 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:08.578424 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:08.578355 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" podStartSLOduration=1.983835883 podStartE2EDuration="4.578336054s" totalImagesPullingTime="2.594500171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:08.576727042 +0000 UTC m=+302.600330942" watchObservedRunningTime="2026-10-30 19:22:08.578336054 +0000 UTC m=+302.601939954" Oct 30 19:22:09.562566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:09.561264 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s6mpf" event={"ID":"67077e2a-7385-4c9c-8f36-ec005a1c2b7d","Type":"ContainerStarted","Data":"5b1c7479524e9fc7d0f12968a0cc83dc5d7551a3a013c922c3da469bf64148d6"} Oct 30 19:22:09.562566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:09.562067 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:09.589285 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:09.588023 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-s6mpf" podStartSLOduration=2.750778223 podStartE2EDuration="5.588002799s" totalImagesPullingTime="2.837224576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:09.586390828 +0000 UTC m=+303.609994726" watchObservedRunningTime="2026-10-30 19:22:09.588002799 +0000 UTC m=+303.611606699" Oct 30 19:22:21.573160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:21.573131 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-s6mpf" Oct 30 19:22:24.376461 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:22:24.376404 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:22:36.624172 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.624139 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl"] Oct 30 19:22:36.629194 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.629173 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.638335 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.637657 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl"] Oct 30 19:22:36.640196 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.640169 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:22:36.640293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.640277 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2z7lq\"" Oct 30 19:22:36.640359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.640344 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:22:36.748853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.748812 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.749020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.748879 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2d4\" (UniqueName: \"kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.749020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.748979 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.749120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.749043 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.850348 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.850307 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.850546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.850364 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2d4\" (UniqueName: \"kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.850546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.850407 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.850546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.850508 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.851093 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.851063 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.853145 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.853115 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.853242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.853161 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.859436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.859393 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2d4\" (UniqueName: \"kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4\") pod \"maas-api-585bcbc4bb-npjpl\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:36.944482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:36.944385 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:37.219885 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:37.219852 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl"] Oct 30 19:22:37.221798 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:22:37.221770 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e804365_db54_41d4_8d00_6c84037fb091.slice/crio-9cedbf45b2e8076a8167c807b509369ae6c6d8bf1045217ec4b3ff66adc926eb WatchSource:0}: Error finding container 9cedbf45b2e8076a8167c807b509369ae6c6d8bf1045217ec4b3ff66adc926eb: Status 404 returned error can't find the container with id 9cedbf45b2e8076a8167c807b509369ae6c6d8bf1045217ec4b3ff66adc926eb Oct 30 19:22:37.689823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:37.689787 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" event={"ID":"9e804365-db54-41d4-8d00-6c84037fb091","Type":"ContainerStarted","Data":"9cedbf45b2e8076a8167c807b509369ae6c6d8bf1045217ec4b3ff66adc926eb"} Oct 30 19:22:38.696901 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:38.696863 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" event={"ID":"9e804365-db54-41d4-8d00-6c84037fb091","Type":"ContainerStarted","Data":"ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb"} Oct 30 19:22:38.697343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:38.696935 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:38.720360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:38.720296 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" podStartSLOduration=1.334434904 podStartE2EDuration="2.720278704s" totalImagesPullingTime="1.3858438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:38.716637618 +0000 UTC m=+332.740241517" watchObservedRunningTime="2026-10-30 19:22:38.720278704 +0000 UTC m=+332.743882602" Oct 30 19:22:39.567560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:39.567529 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-79xzr" Oct 30 19:22:40.568834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:40.568804 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:22:44.706751 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:22:44.706723 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 19:22:54.376900 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:22:54.376863 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:23:00.765965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.765926 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8"] Oct 30 19:23:00.769602 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.769577 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:00.782554 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.782533 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8"] Oct 30 19:23:00.850476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.850448 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbqq\" (UniqueName: \"kubernetes.io/projected/1951dba1-e295-4d07-a375-c9ac2992cd5f-kube-api-access-cfbqq\") pod \"llmisvc-controller-manager-6466c554d8-c6ct8\" (UID: \"1951dba1-e295-4d07-a375-c9ac2992cd5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:00.850604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.850509 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1951dba1-e295-4d07-a375-c9ac2992cd5f-cert\") pod \"llmisvc-controller-manager-6466c554d8-c6ct8\" (UID: \"1951dba1-e295-4d07-a375-c9ac2992cd5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:00.951229 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.951196 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1951dba1-e295-4d07-a375-c9ac2992cd5f-cert\") pod \"llmisvc-controller-manager-6466c554d8-c6ct8\" (UID: \"1951dba1-e295-4d07-a375-c9ac2992cd5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:00.951380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.951280 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbqq\" (UniqueName: \"kubernetes.io/projected/1951dba1-e295-4d07-a375-c9ac2992cd5f-kube-api-access-cfbqq\") pod \"llmisvc-controller-manager-6466c554d8-c6ct8\" (UID: \"1951dba1-e295-4d07-a375-c9ac2992cd5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:00.953790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.953767 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1951dba1-e295-4d07-a375-c9ac2992cd5f-cert\") pod \"llmisvc-controller-manager-6466c554d8-c6ct8\" (UID: \"1951dba1-e295-4d07-a375-c9ac2992cd5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:00.962486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:00.962464 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbqq\" (UniqueName: \"kubernetes.io/projected/1951dba1-e295-4d07-a375-c9ac2992cd5f-kube-api-access-cfbqq\") pod \"llmisvc-controller-manager-6466c554d8-c6ct8\" (UID: \"1951dba1-e295-4d07-a375-c9ac2992cd5f\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:01.082303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:01.082279 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:01.355282 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:01.355252 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8"] Oct 30 19:23:01.357382 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:23:01.357352 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1951dba1_e295_4d07_a375_c9ac2992cd5f.slice/crio-74b89248f536ad9e1a1453e44714b40d0b407ab6ddb6044730ea59bf9e40cd2b WatchSource:0}: Error finding container 74b89248f536ad9e1a1453e44714b40d0b407ab6ddb6044730ea59bf9e40cd2b: Status 404 returned error can't find the container with id 74b89248f536ad9e1a1453e44714b40d0b407ab6ddb6044730ea59bf9e40cd2b Oct 30 19:23:01.803474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:01.803439 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" event={"ID":"1951dba1-e295-4d07-a375-c9ac2992cd5f","Type":"ContainerStarted","Data":"74b89248f536ad9e1a1453e44714b40d0b407ab6ddb6044730ea59bf9e40cd2b"} Oct 30 19:23:02.810606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:02.810564 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" event={"ID":"1951dba1-e295-4d07-a375-c9ac2992cd5f","Type":"ContainerStarted","Data":"75721ad28027e0400e7008a2169471722cdd5f587f7f3362e69ddf2c6ac44b23"} Oct 30 19:23:02.811094 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:02.810704 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:24.376494 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:23:24.376388 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:23:33.817571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:33.817534 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" Oct 30 19:23:33.859211 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:33.859157 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-c6ct8" podStartSLOduration=33.442362911000004 podStartE2EDuration="33.859145984s" totalImagesPullingTime="416.783073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:02.839440135 +0000 UTC m=+356.863044025" watchObservedRunningTime="2026-10-30 19:23:33.859145984 +0000 UTC m=+387.882749883" Oct 30 19:23:33.907377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:33.907341 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b"] Oct 30 19:23:33.907656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:33.907627 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" podUID="bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" containerName="manager" containerID="cri-o://a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5" gracePeriod=30 Oct 30 19:23:34.149353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.149329 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:23:34.230665 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.230621 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd\" (UniqueName: \"kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd\") pod \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " Oct 30 19:23:34.230834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.230708 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert\" (UniqueName: \"kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert\") pod \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\" (UID: \"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7\") " Oct 30 19:23:34.232877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.232846 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert" pod "bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" (UID: "bb0b19bb-d8b5-42a1-9854-ba579b87ffd7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:34.232877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.232854 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd" pod "bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" (UID: "bb0b19bb-d8b5-42a1-9854-ba579b87ffd7"). InnerVolumeSpecName "kube-api-access-vbsbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:34.331405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.331336 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbsbd\" (UniqueName: \"kubernetes.io/projected/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-kube-api-access-vbsbd\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:34.331405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.331363 6511 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:34.958820 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.958781 6511 generic.go:356] "Generic (PLEG): container finished" podID="bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" containerID="a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5" exitCode=0 Oct 30 19:23:34.959297 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.958849 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" Oct 30 19:23:34.959297 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.958854 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" event={"ID":"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7","Type":"ContainerDied","Data":"a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5"} Oct 30 19:23:34.959297 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.958915 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b" event={"ID":"bb0b19bb-d8b5-42a1-9854-ba579b87ffd7","Type":"ContainerDied","Data":"297c2195a4460a2c6014f0e48216e821374740ebd3188ec37ee2fed1ecc7a7e1"} Oct 30 19:23:34.959297 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.958937 6511 scope.go:122] "RemoveContainer" containerID="a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5" Oct 30 19:23:34.968481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.968461 6511 scope.go:122] "RemoveContainer" containerID="a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5" Oct 30 19:23:34.968723 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:23:34.968706 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5\": container with ID starting with a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5 not found: ID does not exist" containerID="a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5" Oct 30 19:23:34.968779 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:34.968731 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5"} err="failed to get container status \"a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5\": rpc error: code = NotFound desc = could not find container \"a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5\": container with ID starting with a919c67f289fb07b93dbe9e08a76df448ee862fd6a18079f06ff5149d9a420a5 not found: ID does not exist" Oct 30 19:23:35.044311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:35.044278 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b"] Oct 30 19:23:35.068377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:35.068345 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xv82b"] Oct 30 19:23:36.100051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:23:36.096576 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" path="/var/lib/kubelet/pods/bb0b19bb-d8b5-42a1-9854-ba579b87ffd7/volumes" Oct 30 19:23:54.377585 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:23:54.377544 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:24:04.101355 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:04.101317 6511 generic.go:356] "Generic (PLEG): container finished" podID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerID="599c2c4487772ce097deaafaf62f1f079e320a5c53bfa22464cbbf3a62474b9d" exitCode=0 Oct 30 19:24:04.101355 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:04.101359 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" event={"ID":"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f","Type":"ContainerDied","Data":"599c2c4487772ce097deaafaf62f1f079e320a5c53bfa22464cbbf3a62474b9d"} Oct 30 19:24:04.101826 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:04.101729 6511 scope.go:122] "RemoveContainer" containerID="599c2c4487772ce097deaafaf62f1f079e320a5c53bfa22464cbbf3a62474b9d" Oct 30 19:24:04.102818 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:04.102789 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:24:04.102902 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:04.102859 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:24:04.102902 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:04.102891 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:24:05.108321 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:05.108285 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" event={"ID":"ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f","Type":"ContainerStarted","Data":"bceb4b1a2935a4ff7ce762dbdfa38eef1a8128bd3305f92a6ab2cad711a00315"} Oct 30 19:24:05.834956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:05.834856 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:24:05.835442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:05.835267 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Oct 30 19:24:05.835442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:05.835323 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Oct 30 19:24:06.834884 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:06.834839 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Oct 30 19:24:06.835536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:06.835500 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Oct 30 19:24:07.836441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:07.835850 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:24:07.836938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:07.836535 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Oct 30 19:24:07.836938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:07.836596 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Oct 30 19:24:08.834147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:08.833737 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Oct 30 19:24:08.834147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:08.833793 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Oct 30 19:24:09.834440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:09.833843 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Oct 30 19:24:09.834440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:09.833918 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Oct 30 19:24:10.833763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:10.833715 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body= Oct 30 19:24:10.833947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:10.833789 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" Oct 30 19:24:12.835076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:12.835026 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:24:12.835613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:12.835099 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:24:13.876617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:13.876567 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:24:13.877100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:13.876649 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:24:14.137232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:14.137132 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:24:14.138224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:24:14.138203 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" Oct 30 19:24:24.377031 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:24:24.376994 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:24:54.376930 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:24:54.376854 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:25:24.378000 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:25:24.377963 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:25:26.603403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.603371 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh"] Oct 30 19:25:26.603926 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.603752 6511 cpu_manager.go:414] "containerMap: removing container" podUID="bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" containerName="manager" Oct 30 19:25:26.603926 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.603765 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" containerName="manager" Oct 30 19:25:26.603926 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.603855 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0b19bb-d8b5-42a1-9854-ba579b87ffd7" containerName="manager" Oct 30 19:25:26.607024 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.607001 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.620233 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.620214 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ce69f3-openshift-default-dockercfg-ktjvt\"" Oct 30 19:25:26.628009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.627984 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh"] Oct 30 19:25:26.730504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730462 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxz7t\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730683 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730528 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730683 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730557 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730683 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730596 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730683 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730633 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730726 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730790 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730834 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.730917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.730866 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832435 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832380 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832490 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832543 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832599 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832642 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832672 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832718 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxz7t\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832763 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.832881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.832787 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.833276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.833247 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.833485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.833294 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.833621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.833330 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.833727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.833578 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.834038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.834011 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.836265 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.836221 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.838131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.838104 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.847475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.847447 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.847633 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.847598 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxz7t\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.856995 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.856937 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb"] Oct 30 19:25:26.862050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.862034 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.871389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.871369 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ae63bf-openshift-default-dockercfg-2n7kg\"" Oct 30 19:25:26.873988 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.873967 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb"] Oct 30 19:25:26.919846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.919800 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:26.934153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934128 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934192 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934278 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934300 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrjj\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934351 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934368 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934392 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934425 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:26.934471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:26.934466 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036075 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036165 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036195 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrjj\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036276 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036299 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036326 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036351 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036392 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.036468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.036443 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.037225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.037193 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.037344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.037311 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.037608 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.037583 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.037909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.037859 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.038098 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.038070 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.039497 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.039471 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.040431 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.040396 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.046298 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.046274 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.046376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.046349 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrjj\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.176809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.176773 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:27.212671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.212438 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh"] Oct 30 19:25:27.213839 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:25:27.213804 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32977db8_7d4d_4fdd_a156_f87fb7cdd079.slice/crio-bce4d70a6940cc8788efbd249c693df1474e269dcd3ad347a98437bb1bb88009 WatchSource:0}: Error finding container bce4d70a6940cc8788efbd249c693df1474e269dcd3ad347a98437bb1bb88009: Status 404 returned error can't find the container with id bce4d70a6940cc8788efbd249c693df1474e269dcd3ad347a98437bb1bb88009 Oct 30 19:25:27.216184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.216148 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:27.216288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.216228 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:27.216288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.216273 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:27.467674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.467645 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb"] Oct 30 19:25:27.470460 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:25:27.470394 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571df9c4_0659_4745_914b_8f36babc24e8.slice/crio-457bf28bd30820f38c06f022f9cd95777dd73a05fc273e72ac51041b1cef13af WatchSource:0}: Error finding container 457bf28bd30820f38c06f022f9cd95777dd73a05fc273e72ac51041b1cef13af: Status 404 returned error can't find the container with id 457bf28bd30820f38c06f022f9cd95777dd73a05fc273e72ac51041b1cef13af Oct 30 19:25:27.473057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.473028 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:27.473132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.473089 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:27.473132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.473119 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:27.511970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.511940 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" event={"ID":"571df9c4-0659-4745-914b-8f36babc24e8","Type":"ContainerStarted","Data":"457bf28bd30820f38c06f022f9cd95777dd73a05fc273e72ac51041b1cef13af"} Oct 30 19:25:27.513892 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.513863 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" event={"ID":"32977db8-7d4d-4fdd-a156-f87fb7cdd079","Type":"ContainerStarted","Data":"6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca"} Oct 30 19:25:27.514026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.513926 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" event={"ID":"32977db8-7d4d-4fdd-a156-f87fb7cdd079","Type":"ContainerStarted","Data":"bce4d70a6940cc8788efbd249c693df1474e269dcd3ad347a98437bb1bb88009"} Oct 30 19:25:27.542013 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.541958 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podStartSLOduration=1.541938301 podStartE2EDuration="1.541938301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:27.539538575 +0000 UTC m=+501.563142476" watchObservedRunningTime="2026-10-30 19:25:27.541938301 +0000 UTC m=+501.565542203" Oct 30 19:25:27.920593 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:27.920553 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:28.522053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:28.522007 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" event={"ID":"571df9c4-0659-4745-914b-8f36babc24e8","Type":"ContainerStarted","Data":"c165bcc42a53926f204f025fc5175da4384ec82713fd864d38aba3fd89fadd14"} Oct 30 19:25:28.552279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:28.552231 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podStartSLOduration=2.552217659 podStartE2EDuration="2.552217659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:28.549574202 +0000 UTC m=+502.573178101" watchObservedRunningTime="2026-10-30 19:25:28.552217659 +0000 UTC m=+502.575821557" Oct 30 19:25:28.926060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:28.926029 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:29.177175 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:29.177098 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:29.181922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:29.181898 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:29.527701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:29.527615 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:29.527701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:29.527675 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:29.528304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:29.528281 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:29.528390 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:29.528340 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:37.174430 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.174383 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp"] Oct 30 19:25:37.178252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.178230 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.192497 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.192375 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-419d0a-serving-cert\"" Oct 30 19:25:37.194545 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.194524 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp"] Oct 30 19:25:37.243474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.243061 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.243474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.243168 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.243474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.243206 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpp5z\" (UniqueName: \"kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.243474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.243239 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.343970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.343934 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.344167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.344022 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.344167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.344053 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpp5z\" (UniqueName: \"kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.344167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.344086 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.345101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.345064 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.346755 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.346733 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.346832 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.346733 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.353812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.353788 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpp5z\" (UniqueName: \"kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z\") pod \"maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.492035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.491941 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:37.802153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.802125 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp"] Oct 30 19:25:37.804780 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:25:37.804751 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73793a2_8c80_4122_b5d9_36a628bec9f1.slice/crio-cc8f02879510de3ec6e71946b9f6ee6fd8b515dcdf5b62d62844fe4558c36d38 WatchSource:0}: Error finding container cc8f02879510de3ec6e71946b9f6ee6fd8b515dcdf5b62d62844fe4558c36d38: Status 404 returned error can't find the container with id cc8f02879510de3ec6e71946b9f6ee6fd8b515dcdf5b62d62844fe4558c36d38 Oct 30 19:25:37.806083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.806064 6511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:25:37.983919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.983887 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5"] Oct 30 19:25:37.988675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.988648 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:37.996930 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:37.996905 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-b623ca-serving-cert\"" Oct 30 19:25:38.000197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.000170 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5"] Oct 30 19:25:38.051084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.051045 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.051278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.051108 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.051278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.051218 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqncx\" (UniqueName: \"kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.051278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.051255 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.151913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.151832 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.151913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.151874 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqncx\" (UniqueName: \"kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.151913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.151892 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.152195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.151969 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.152607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.152584 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.154642 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.154617 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.154754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.154734 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.161761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.161742 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqncx\" (UniqueName: \"kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx\") pod \"maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.305148 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.305097 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:38.592553 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.592509 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" event={"ID":"c73793a2-8c80-4122-b5d9-36a628bec9f1","Type":"ContainerStarted","Data":"01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841"} Oct 30 19:25:38.592732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.592606 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" event={"ID":"c73793a2-8c80-4122-b5d9-36a628bec9f1","Type":"ContainerStarted","Data":"cc8f02879510de3ec6e71946b9f6ee6fd8b515dcdf5b62d62844fe4558c36d38"} Oct 30 19:25:38.592732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.592634 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:38.645705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.645659 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5"] Oct 30 19:25:38.646860 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:25:38.646833 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd6751c_1408_4557_82bc_d9e581864563.slice/crio-e2386dc2b8fc953da6024d3abcbb7525ef0cf508544f276111f4a7ecac3963bd WatchSource:0}: Error finding container e2386dc2b8fc953da6024d3abcbb7525ef0cf508544f276111f4a7ecac3963bd: Status 404 returned error can't find the container with id e2386dc2b8fc953da6024d3abcbb7525ef0cf508544f276111f4a7ecac3963bd Oct 30 19:25:38.680265 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:38.680165 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" podStartSLOduration=1.083981302 podStartE2EDuration="1.680145136s" totalImagesPullingTime="596.163834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:38.678754497 +0000 UTC m=+512.702358409" watchObservedRunningTime="2026-10-30 19:25:38.680145136 +0000 UTC m=+512.703749040" Oct 30 19:25:39.599323 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.599288 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" event={"ID":"bdd6751c-1408-4557-82bc-d9e581864563","Type":"ContainerStarted","Data":"a5db530d0a4d42f0a00f24e025907010784c00b3b38e9c539535f1a2a8eed5a7"} Oct 30 19:25:39.599774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.599344 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" event={"ID":"bdd6751c-1408-4557-82bc-d9e581864563","Type":"ContainerStarted","Data":"e2386dc2b8fc953da6024d3abcbb7525ef0cf508544f276111f4a7ecac3963bd"} Oct 30 19:25:39.599774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.599599 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:39.681110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.681049 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" podStartSLOduration=2.237120917 podStartE2EDuration="2.681032333s" totalImagesPullingTime="443.911416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:39.623573086 +0000 UTC m=+513.647176987" watchObservedRunningTime="2026-10-30 19:25:39.681032333 +0000 UTC m=+513.704636246" Oct 30 19:25:39.682330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.682306 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk"] Oct 30 19:25:39.685965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.685947 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.702716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.702382 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ae63bf-serving-cert\"" Oct 30 19:25:39.707760 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.707732 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk"] Oct 30 19:25:39.769330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.769298 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.769527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.769349 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.769527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.769486 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.769611 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.769529 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkgs\" (UniqueName: \"kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.871034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.870951 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.871034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.871010 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.871221 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.871049 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkgs\" (UniqueName: \"kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.871221 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.871130 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.871848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.871824 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.873682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.873657 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.873784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.873735 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:39.880255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:39.880233 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkgs\" (UniqueName: \"kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs\") pod \"maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:40.000913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:40.000873 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:40.305288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:40.305259 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk"] Oct 30 19:25:40.605748 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:40.605643 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" event={"ID":"715bf9e2-bae1-4b7f-a55e-b789286ad80b","Type":"ContainerStarted","Data":"d3d9978ade83a0173681bd2d6bb0066a24c88174f9123bafd725693ce0754670"} Oct 30 19:25:41.611985 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:41.611951 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" event={"ID":"715bf9e2-bae1-4b7f-a55e-b789286ad80b","Type":"ContainerStarted","Data":"dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561"} Oct 30 19:25:41.612453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:41.612078 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:41.643466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:41.640675 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" podStartSLOduration=2.167409092 podStartE2EDuration="2.64065515s" totalImagesPullingTime="473.246058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:41.637823852 +0000 UTC m=+515.661427754" watchObservedRunningTime="2026-10-30 19:25:41.64065515 +0000 UTC m=+515.664259055" Oct 30 19:25:44.604613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:44.604585 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:25:45.536406 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.535406 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:25:45.536406 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.535488 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:25:45.611686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.611665 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:25:45.639031 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.638996 6511 generic.go:356] "Generic (PLEG): container finished" podID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerID="6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca" exitCode=0 Oct 30 19:25:45.639240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.639072 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" event={"ID":"32977db8-7d4d-4fdd-a156-f87fb7cdd079","Type":"ContainerDied","Data":"6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca"} Oct 30 19:25:45.639465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.639448 6511 scope.go:122] "RemoveContainer" containerID="6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca" Oct 30 19:25:45.640780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.640743 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:45.640889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.640818 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:45.640889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:45.640876 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:46.677086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:46.677056 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" event={"ID":"32977db8-7d4d-4fdd-a156-f87fb7cdd079","Type":"ContainerStarted","Data":"60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2"} Oct 30 19:25:46.922441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:46.921790 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:46.922441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:46.921924 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:25:46.922441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:46.921967 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 19:25:47.685655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:47.685614 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:25:47.922015 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:47.921977 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:25:47.922319 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:47.922270 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 19:25:48.923982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:48.923943 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:25:48.924623 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:48.924570 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 19:25:49.921833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:49.921796 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:25:49.922125 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:49.922092 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 19:25:50.923453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:50.921839 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 19:25:50.923453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:50.921914 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 19:25:51.961466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:51.961431 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:51.962026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:51.961853 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:51.962788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:51.962768 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:25:54.380234 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:25:54.380195 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:25:55.735285 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:55.735254 6511 generic.go:356] "Generic (PLEG): container finished" podID="571df9c4-0659-4745-914b-8f36babc24e8" containerID="c165bcc42a53926f204f025fc5175da4384ec82713fd864d38aba3fd89fadd14" exitCode=0 Oct 30 19:25:55.735722 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:55.735324 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" event={"ID":"571df9c4-0659-4745-914b-8f36babc24e8","Type":"ContainerDied","Data":"c165bcc42a53926f204f025fc5175da4384ec82713fd864d38aba3fd89fadd14"} Oct 30 19:25:55.735765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:55.735724 6511 scope.go:122] "RemoveContainer" containerID="c165bcc42a53926f204f025fc5175da4384ec82713fd864d38aba3fd89fadd14" Oct 30 19:25:55.737001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:55.736969 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:55.737097 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:55.737041 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:55.737097 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:55.737072 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:25:56.752442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:56.752306 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" event={"ID":"571df9c4-0659-4745-914b-8f36babc24e8","Type":"ContainerStarted","Data":"356d2a1cfec1a88bed27d05f6ca37057b400273508677404dc99686b1a54483c"} Oct 30 19:25:57.177016 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:57.176969 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:25:57.179400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:57.179362 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:25:57.179574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:57.179445 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 19:25:58.177364 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:58.177325 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:25:58.177874 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:58.177389 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 19:25:59.177150 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:59.177114 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:25:59.177328 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:59.177170 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 19:25:59.528494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:25:59.527528 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:26:00.177346 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:00.177305 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:26:00.177578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:00.177377 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 19:26:01.177207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:01.177168 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Oct 30 19:26:01.177700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:01.177228 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Oct 30 19:26:02.181256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:02.181221 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:26:02.182344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:02.182317 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:26:24.378770 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:26:24.378687 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:26:28.687617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.687583 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g"] Oct 30 19:26:28.691398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.691374 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.700083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.700061 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"premium-w5-60577a-kserve-self-signed-certs\"" Oct 30 19:26:28.704358 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.704341 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"default-dockercfg-w4x7t\"" Oct 30 19:26:28.705347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.705251 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"openshift-service-ca.crt\"" Oct 30 19:26:28.705567 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.705549 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"kube-root-ca.crt\"" Oct 30 19:26:28.707457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.707437 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g"] Oct 30 19:26:28.722065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.722034 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.722065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.722063 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.722226 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.722093 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.722308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.722283 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.722373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.722322 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwl7\" (UniqueName: \"kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.722448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.722435 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824291 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824343 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824386 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824507 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824537 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwl7\" (UniqueName: \"kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824768 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824578 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.824942 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824920 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.825035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.824980 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.825093 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.825061 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.832448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.827443 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.832448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.828038 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:28.838621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:28.838592 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwl7\" (UniqueName: \"kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7\") pod \"premium-w5-60577a-kserve-6ff5bddc9f-gwg6g\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:29.005118 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:29.005043 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:29.288354 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:29.288272 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g"] Oct 30 19:26:29.290938 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:26:29.290905 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94797435_fb10_46aa_b4ad_4c1c7deee1cb.slice/crio-762afe2a5a8503b5550f26da0322f7672ebc0bb56395d4a640c9260f6e18d02d WatchSource:0}: Error finding container 762afe2a5a8503b5550f26da0322f7672ebc0bb56395d4a640c9260f6e18d02d: Status 404 returned error can't find the container with id 762afe2a5a8503b5550f26da0322f7672ebc0bb56395d4a640c9260f6e18d02d Oct 30 19:26:29.927996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:29.927953 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" event={"ID":"94797435-fb10-46aa-b4ad-4c1c7deee1cb","Type":"ContainerStarted","Data":"762afe2a5a8503b5550f26da0322f7672ebc0bb56395d4a640c9260f6e18d02d"} Oct 30 19:26:30.935214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:30.935179 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" event={"ID":"94797435-fb10-46aa-b4ad-4c1c7deee1cb","Type":"ContainerStarted","Data":"87a93e18ec5d15453938119ec15767812826ad8ed06c765dcf99c081c6ea5af2"} Oct 30 19:26:30.954369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:30.954323 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podStartSLOduration=1.729122058 podStartE2EDuration="2.954307508s" totalImagesPullingTime="1.22518545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:30.954156863 +0000 UTC m=+564.977760771" watchObservedRunningTime="2026-10-30 19:26:30.954307508 +0000 UTC m=+564.977911407" Oct 30 19:26:31.006096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:31.006071 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:39.005879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:39.005843 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:39.018511 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:39.018484 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:39.994127 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:39.994098 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:26:54.377851 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:26:54.377814 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:26:56.468384 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.468352 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p"] Oct 30 19:26:56.471888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.471869 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.478762 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.478670 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-60577a-openshift-default-dockercfg-86d5l\"" Oct 30 19:26:56.492378 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.492356 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p"] Oct 30 19:26:56.572433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572388 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572454 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572540 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572568 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572731 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572585 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572731 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572644 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78r4v\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572731 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572700 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572781 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.572830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.572804 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674215 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674183 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78r4v\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674228 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674264 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674295 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674618 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674469 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674618 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674520 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674618 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674580 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674611 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674664 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674786 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.674904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.674861 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.675123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.675102 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.675326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.675305 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.675435 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.675400 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.677254 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.677232 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.677397 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.677381 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.686049 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.686025 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78r4v\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.688233 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.688212 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.784553 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.784485 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:56.952506 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.952468 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2"] Oct 30 19:26:56.957966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.957937 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:56.972985 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.972962 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad96a4-openshift-default-dockercfg-6zjhk\"" Oct 30 19:26:56.974373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:56.974350 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2"] Oct 30 19:26:57.068001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.067976 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p"] Oct 30 19:26:57.069513 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:26:57.069487 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda037d7c5_52cf_495b_967b_792c42da4ce6.slice/crio-d2a2b45351f31f706645e86a0a0bd16b93c492bb865c2a0530753d90a21805c0 WatchSource:0}: Error finding container d2a2b45351f31f706645e86a0a0bd16b93c492bb865c2a0530753d90a21805c0: Status 404 returned error can't find the container with id d2a2b45351f31f706645e86a0a0bd16b93c492bb865c2a0530753d90a21805c0 Oct 30 19:26:57.071718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.071680 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:26:57.071842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.071760 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:26:57.071842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.071805 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:26:57.078291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.078267 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" event={"ID":"a037d7c5-52cf-495b-967b-792c42da4ce6","Type":"ContainerStarted","Data":"d2a2b45351f31f706645e86a0a0bd16b93c492bb865c2a0530753d90a21805c0"} Oct 30 19:26:57.079711 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079687 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6db6r\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.079813 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079724 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.079813 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079749 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.079912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079831 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.079912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079903 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.079990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079922 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.079990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.079957 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.080083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.080009 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.080083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.080077 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.180896 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.180865 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6db6r\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.180907 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.180941 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.180969 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.180996 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181017 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181075 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181106 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181159 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181311 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181610 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181524 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181648 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.181893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.181874 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.182304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.182283 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.183835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.183807 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.183932 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.183911 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.190051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.190026 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.190247 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.190224 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6db6r\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.271211 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.271175 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:57.558382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.558351 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2"] Oct 30 19:26:57.562590 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:26:57.562537 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d4dd33_de2d_44d3_8676_1a6a940d100f.slice/crio-159b5cf1d33ac9b36354cb0491e7a1a2af55e356d8a9b3849256a7b16483ed22 WatchSource:0}: Error finding container 159b5cf1d33ac9b36354cb0491e7a1a2af55e356d8a9b3849256a7b16483ed22: Status 404 returned error can't find the container with id 159b5cf1d33ac9b36354cb0491e7a1a2af55e356d8a9b3849256a7b16483ed22 Oct 30 19:26:57.565115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.565085 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:26:57.565194 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.565152 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:26:57.565194 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:57.565182 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:26:58.089322 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.089290 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" event={"ID":"38d4dd33-de2d-44d3-8676-1a6a940d100f","Type":"ContainerStarted","Data":"214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2"} Oct 30 19:26:58.089525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.089343 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" event={"ID":"38d4dd33-de2d-44d3-8676-1a6a940d100f","Type":"ContainerStarted","Data":"159b5cf1d33ac9b36354cb0491e7a1a2af55e356d8a9b3849256a7b16483ed22"} Oct 30 19:26:58.097828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.097756 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" event={"ID":"a037d7c5-52cf-495b-967b-792c42da4ce6","Type":"ContainerStarted","Data":"3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039"} Oct 30 19:26:58.121437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.117881 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podStartSLOduration=2.117864221 podStartE2EDuration="2.117864221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:58.115237478 +0000 UTC m=+592.138841377" watchObservedRunningTime="2026-10-30 19:26:58.117864221 +0000 UTC m=+592.141468121" Oct 30 19:26:58.271711 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.271666 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:26:58.273401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.273374 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:58.273544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.273439 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:26:58.785555 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.785517 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:26:58.786891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.786870 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:58.787036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:58.787012 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:26:59.272177 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:59.272139 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:59.272389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:59.272207 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:26:59.785560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:59.785520 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:59.785930 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:26:59.785577 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:27:00.274893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:00.274855 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:27:00.275202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:00.275162 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:27:00.784916 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:00.784887 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:27:00.785135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:00.785110 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:27:01.271573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:01.271537 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:27:01.272086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:01.271586 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:27:01.785188 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:01.785035 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:27:01.785188 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:01.785092 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:27:02.272108 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:02.272064 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:27:02.272662 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:02.272117 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:27:02.784877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:02.784842 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:27:02.785068 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:02.784899 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 19:27:03.272584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:03.272550 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:27:03.272997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:03.272611 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 19:27:04.023594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.023553 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:27:04.024716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.024688 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:27:04.026022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.025999 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:27:04.049851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.049803 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podStartSLOduration=8.049789984 podStartE2EDuration="8.049789984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:58.145061944 +0000 UTC m=+592.168665843" watchObservedRunningTime="2026-10-30 19:27:04.049789984 +0000 UTC m=+598.073393882" Oct 30 19:27:04.326514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.326483 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:27:04.327000 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.326901 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:27:04.327782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:04.327750 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:27:06.179055 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:06.179028 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:27:06.182441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:06.182397 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:27:11.197726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.197696 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx"] Oct 30 19:27:11.203979 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.203956 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.211592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.211572 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-01b24e-openshift-default-dockercfg-tz4v8\"" Oct 30 19:27:11.218491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.218465 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx"] Oct 30 19:27:11.257706 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.257677 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.257831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.257731 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.257831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.257804 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.257917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.257847 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.257917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.257875 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.258029 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.258007 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.258081 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.258069 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.258112 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.258090 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n69n\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.258157 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.258115 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359464 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359401 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359483 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359512 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359551 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359576 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n69n\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359806 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359611 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359806 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359792 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359822 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359856 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359915 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.359953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359949 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.360247 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.359974 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.360247 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.360226 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.360515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.360492 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.361997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.361976 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.362365 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.362350 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.368842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.368823 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.369198 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.369178 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n69n\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.517625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.517544 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:11.618079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.618050 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb"] Oct 30 19:27:11.622320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.622303 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.633144 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.633124 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-419d0a-openshift-default-dockercfg-tghgq\"" Oct 30 19:27:11.636752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.636732 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb"] Oct 30 19:27:11.662108 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662083 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662132 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hcw\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662240 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662277 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662305 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662392 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662482 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662532 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662523 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.662565 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.662553 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.763644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763609 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.763644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763649 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.763854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763713 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.763854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763753 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hcw\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.763854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763803 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.763854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763837 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.763864 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764024 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764094 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764273 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764344 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764538 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764501 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764546 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.764621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.764581 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.766309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.766290 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.766460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.766443 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.773780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.773720 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.773780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.773749 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hcw\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:11.799658 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.799636 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx"] Oct 30 19:27:11.800243 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:27:11.800223 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d505aa_9ea6_4e1a_b3b8_d9da39b8b448.slice/crio-5f3fe9f7e68869ff4a156a6fa5bc2fbe3a7a7aa754f3f7b7bf21e7f6d280d9ff WatchSource:0}: Error finding container 5f3fe9f7e68869ff4a156a6fa5bc2fbe3a7a7aa754f3f7b7bf21e7f6d280d9ff: Status 404 returned error can't find the container with id 5f3fe9f7e68869ff4a156a6fa5bc2fbe3a7a7aa754f3f7b7bf21e7f6d280d9ff Oct 30 19:27:11.802700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.802665 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:11.802764 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.802732 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:11.802811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.802768 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:11.936169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:11.936140 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:12.195049 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.195014 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" event={"ID":"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448","Type":"ContainerStarted","Data":"f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55"} Oct 30 19:27:12.195259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.195070 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" event={"ID":"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448","Type":"ContainerStarted","Data":"5f3fe9f7e68869ff4a156a6fa5bc2fbe3a7a7aa754f3f7b7bf21e7f6d280d9ff"} Oct 30 19:27:12.219282 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.219233 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podStartSLOduration=1.219219096 podStartE2EDuration="1.219219096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:12.217288457 +0000 UTC m=+606.240892360" watchObservedRunningTime="2026-10-30 19:27:12.219219096 +0000 UTC m=+606.242822994" Oct 30 19:27:12.231752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.231729 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb"] Oct 30 19:27:12.234098 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:27:12.234063 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f52211_5af3_426c_8be3_c486c14bd2dc.slice/crio-bf08152dd60b4c05618fea335cf499cf593d4fb61c13f339c3d0555aec60ebb9 WatchSource:0}: Error finding container bf08152dd60b4c05618fea335cf499cf593d4fb61c13f339c3d0555aec60ebb9: Status 404 returned error can't find the container with id bf08152dd60b4c05618fea335cf499cf593d4fb61c13f339c3d0555aec60ebb9 Oct 30 19:27:12.236539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.236505 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:12.236644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.236588 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:12.236644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.236637 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:12.523494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.523009 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:12.523494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.523139 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:12.523494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:12.523185 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:27:13.201676 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.201640 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" event={"ID":"30f52211-5af3-426c-8be3-c486c14bd2dc","Type":"ContainerStarted","Data":"7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5"} Oct 30 19:27:13.201956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.201933 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" event={"ID":"30f52211-5af3-426c-8be3-c486c14bd2dc","Type":"ContainerStarted","Data":"bf08152dd60b4c05618fea335cf499cf593d4fb61c13f339c3d0555aec60ebb9"} Oct 30 19:27:13.244571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.244510 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podStartSLOduration=2.244489372 podStartE2EDuration="2.244489372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:13.237983208 +0000 UTC m=+607.261587120" watchObservedRunningTime="2026-10-30 19:27:13.244489372 +0000 UTC m=+607.268093272" Oct 30 19:27:13.522308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.521790 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:13.522308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.521882 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:27:13.939302 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.939269 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:13.939686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.939662 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:13.939854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:13.939818 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:27:14.529442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:14.521845 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:14.529442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:14.521917 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:27:14.936861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:14.936820 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:14.937177 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:14.937142 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:27:15.532817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:15.532778 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:15.533476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:15.533444 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:27:15.947441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:15.946810 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:15.947441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:15.946876 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:27:16.539441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:16.538554 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:16.539441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:16.538631 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:27:16.941819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:16.941781 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:16.942112 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:16.942084 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:27:17.522203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:17.521771 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:17.522203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:17.521835 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 19:27:17.937802 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:17.937763 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:17.938381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:17.938348 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 19:27:18.585458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:18.585426 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:18.585932 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:18.585907 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:18.587114 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:18.587094 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:27:19.041655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:19.041575 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:19.042485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:19.042463 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:19.043690 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:19.043650 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:27:24.384395 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:27:24.384355 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:27:26.315632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.315600 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr"] Oct 30 19:27:26.319269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.319249 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.327290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.327268 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b623ca-openshift-default-dockercfg-9vknm\"" Oct 30 19:27:26.333319 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.333295 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr"] Oct 30 19:27:26.449198 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449163 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449207 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449322 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449373 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449476 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449556 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449529 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7tp\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449565 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449588 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.449663 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.449631 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550669 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550718 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7tp\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550741 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550773 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550793 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550822 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.550879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.550867 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551063 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551134 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551150 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551154 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551208 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551380 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.551725 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.551700 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.553356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.553331 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.553475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.553395 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.559649 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.559626 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.559716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.559685 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7tp\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.633526 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.633445 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:26.918050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.918020 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr"] Oct 30 19:27:26.919810 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:27:26.919786 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abf50bf_3416_46b3_b084_796963443d56.slice/crio-94f71f12f8c26643f568a4e1b8644cb204910faff4e475f047e7b93e54edc956 WatchSource:0}: Error finding container 94f71f12f8c26643f568a4e1b8644cb204910faff4e475f047e7b93e54edc956: Status 404 returned error can't find the container with id 94f71f12f8c26643f568a4e1b8644cb204910faff4e475f047e7b93e54edc956 Oct 30 19:27:26.922184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.922152 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:26.922329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.922223 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:26.922382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:26.922368 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:27:27.332224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:27.332187 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" event={"ID":"5abf50bf-3416-46b3-b084-796963443d56","Type":"ContainerStarted","Data":"d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda"} Oct 30 19:27:27.332675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:27.332244 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" event={"ID":"5abf50bf-3416-46b3-b084-796963443d56","Type":"ContainerStarted","Data":"94f71f12f8c26643f568a4e1b8644cb204910faff4e475f047e7b93e54edc956"} Oct 30 19:27:27.642590 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:27.642547 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:27.644242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:27.644214 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:27.644398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:27.644264 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:28.641436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:28.640863 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:28.641436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:28.640927 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:29.639517 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:29.639481 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:29.639805 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:29.639774 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:30.636805 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:30.636767 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:30.637386 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:30.637352 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:31.636443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:31.634787 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:31.636443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:31.634848 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:32.635429 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:32.634983 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:32.635429 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:32.635046 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:33.634809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:33.634762 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:27:33.634986 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:33.634859 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 19:27:34.638872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:34.638843 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:34.639324 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:34.639234 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:34.639866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:34.639848 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:27:34.662796 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:34.662750 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podStartSLOduration=8.662738476 podStartE2EDuration="8.662738476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:27.36106557 +0000 UTC m=+621.384669472" watchObservedRunningTime="2026-10-30 19:27:34.662738476 +0000 UTC m=+628.686342374" Oct 30 19:27:49.275975 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.275889 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94"] Oct 30 19:27:49.283054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.283000 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.288956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.288924 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94"] Oct 30 19:27:49.295727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.295687 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"kube-root-ca.crt\"" Oct 30 19:27:49.295908 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.295776 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"openshift-service-ca.crt\"" Oct 30 19:27:49.297035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.297010 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"default-dockercfg-cvp5n\"" Oct 30 19:27:49.297154 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.297104 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"distinct-w1-419d0a-kserve-self-signed-certs\"" Oct 30 19:27:49.336797 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.336763 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.336976 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.336816 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.336976 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.336945 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwq4x\" (UniqueName: \"kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.337077 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.337009 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.337161 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.337138 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.337268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.337250 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.441522 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.441657 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.441737 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.441765 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.441807 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwq4x\" (UniqueName: \"kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.441863 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.443438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.442450 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.447828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.445395 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.447828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.445678 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.447828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.447716 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.462090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.448557 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.475231 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.468065 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwq4x\" (UniqueName: \"kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x\") pod \"distinct-w1-419d0a-kserve-cd5dc97cc-jrp94\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.603684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.603635 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:49.890197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:49.890168 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94"] Oct 30 19:27:49.892220 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:27:49.892191 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6ad163f_a529_4148_8a0c_da13da341853.slice/crio-1984d6245215403a2ed5d6fd3a925505bd830a0efda65920525a01bf1ca41101 WatchSource:0}: Error finding container 1984d6245215403a2ed5d6fd3a925505bd830a0efda65920525a01bf1ca41101: Status 404 returned error can't find the container with id 1984d6245215403a2ed5d6fd3a925505bd830a0efda65920525a01bf1ca41101 Oct 30 19:27:50.503350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:50.503318 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" event={"ID":"c6ad163f-a529-4148-8a0c-da13da341853","Type":"ContainerStarted","Data":"3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1"} Oct 30 19:27:50.503350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:50.503357 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" event={"ID":"c6ad163f-a529-4148-8a0c-da13da341853","Type":"ContainerStarted","Data":"1984d6245215403a2ed5d6fd3a925505bd830a0efda65920525a01bf1ca41101"} Oct 30 19:27:50.524223 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:50.524177 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podStartSLOduration=1.524164094 podStartE2EDuration="1.524164094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:50.522715043 +0000 UTC m=+644.546318942" watchObservedRunningTime="2026-10-30 19:27:50.524164094 +0000 UTC m=+644.547767993" Oct 30 19:27:50.604559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:50.604531 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:54.377183 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:27:54.377138 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:27:59.618246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:59.618215 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:27:59.689303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:27:59.688867 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:28:00.914070 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:00.914035 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:28:01.897267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.897229 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn"] Oct 30 19:28:01.902783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.902754 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:01.913064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.912738 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"openshift-service-ca.crt\"" Oct 30 19:28:01.913064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.912836 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"default-dockercfg-tdws5\"" Oct 30 19:28:01.913064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.912930 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"distinct-w6-01b24e-kserve-self-signed-certs\"" Oct 30 19:28:01.913064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.913005 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-01b24e\"/\"kube-root-ca.crt\"" Oct 30 19:28:01.917313 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:01.917285 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn"] Oct 30 19:28:02.037094 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.037055 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n796c\" (UniqueName: \"kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.037566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.037537 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.037780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.037759 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.038020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.038003 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.038336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.038304 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.038560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.038543 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.140364 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.140326 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.140578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.140434 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.140578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.140476 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.140578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.140554 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n796c\" (UniqueName: \"kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.140744 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.140596 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.140744 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.140621 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.141101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.141072 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.141398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.141374 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.141553 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.141530 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.143727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.143701 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.143830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.143816 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.151345 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.151291 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n796c\" (UniqueName: \"kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c\") pod \"distinct-w6-01b24e-kserve-64fbfd9c75-mvttn\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.222678 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.222646 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:02.531301 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.531268 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn"] Oct 30 19:28:02.532718 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:02.532692 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc30068df_e7f9_401d_8da8_cd343a8581f4.slice/crio-4539c9bdfcfb4ce06a227cfa640a3e6cf49ac76271fbb03c59d0ca83f55517fb WatchSource:0}: Error finding container 4539c9bdfcfb4ce06a227cfa640a3e6cf49ac76271fbb03c59d0ca83f55517fb: Status 404 returned error can't find the container with id 4539c9bdfcfb4ce06a227cfa640a3e6cf49ac76271fbb03c59d0ca83f55517fb Oct 30 19:28:02.674027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.673995 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" event={"ID":"c30068df-e7f9-401d-8da8-cd343a8581f4","Type":"ContainerStarted","Data":"0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e"} Oct 30 19:28:02.674186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.674036 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" event={"ID":"c30068df-e7f9-401d-8da8-cd343a8581f4","Type":"ContainerStarted","Data":"4539c9bdfcfb4ce06a227cfa640a3e6cf49ac76271fbb03c59d0ca83f55517fb"} Oct 30 19:28:02.695640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:02.695593 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podStartSLOduration=1.695579019 podStartE2EDuration="1.695579019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:02.694725093 +0000 UTC m=+656.718328993" watchObservedRunningTime="2026-10-30 19:28:02.695579019 +0000 UTC m=+656.719182917" Oct 30 19:28:03.223659 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:03.223626 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:11.452660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.452625 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx"] Oct 30 19:28:11.459706 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.459677 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.478547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.478519 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx"] Oct 30 19:28:11.556898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.556855 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.556920 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557014 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557051 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557082 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557181 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557239 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557225 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557274 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557264 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.557306 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.557291 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbv8\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.658376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.658341 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.658585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.658406 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.658585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.658479 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.658585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.658526 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.658585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.658566 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659003 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.658593 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659060 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659088 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659362 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659140 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbv8\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659362 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659242 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659362 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659264 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659362 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659347 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659548 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.659700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.659642 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.661082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.661060 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.661288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.661269 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.674663 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.674637 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbv8\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.677160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.677137 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:11.782107 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:11.782014 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:12.069304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.069213 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx"] Oct 30 19:28:12.071982 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:12.071955 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20f0cec_20fa_4300_8838_18f0ef03e9dc.slice/crio-a62bb7a4d17a031b0b3924442f60f4553e00b2117303c5ffda77c7cbdcb3773f WatchSource:0}: Error finding container a62bb7a4d17a031b0b3924442f60f4553e00b2117303c5ffda77c7cbdcb3773f: Status 404 returned error can't find the container with id a62bb7a4d17a031b0b3924442f60f4553e00b2117303c5ffda77c7cbdcb3773f Oct 30 19:28:12.074667 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.074628 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:12.074773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.074724 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:12.074827 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.074768 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:12.223779 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.223735 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:12.237136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.237108 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:12.261189 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.261151 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7"] Oct 30 19:28:12.266907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.266861 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.275640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.275462 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"default-dockercfg-dlbkj\"" Oct 30 19:28:12.275640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.275565 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"kube-root-ca.crt\"" Oct 30 19:28:12.275640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.275583 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"distinct-w2-b623ca-kserve-self-signed-certs\"" Oct 30 19:28:12.275640 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.275614 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"openshift-service-ca.crt\"" Oct 30 19:28:12.277782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.277750 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7"] Oct 30 19:28:12.367721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.367619 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.367721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.367685 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s76p\" (UniqueName: \"kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.367970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.367809 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.367970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.367874 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.368084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.368062 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.368144 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.368113 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.469506 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.469462 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470072 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470231 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470136 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470231 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470189 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s76p\" (UniqueName: \"kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470231 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470222 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470284 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470731 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470857 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470833 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.470994 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.470976 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.473255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.473228 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.473529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.473509 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.481580 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.481559 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s76p\" (UniqueName: \"kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p\") pod \"distinct-w2-b623ca-kserve-c4559454d-kg5c7\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.591218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.591181 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:12.732662 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.732610 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" event={"ID":"d20f0cec-20fa-4300-8838-18f0ef03e9dc","Type":"ContainerStarted","Data":"1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec"} Oct 30 19:28:12.732847 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.732680 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" event={"ID":"d20f0cec-20fa-4300-8838-18f0ef03e9dc","Type":"ContainerStarted","Data":"a62bb7a4d17a031b0b3924442f60f4553e00b2117303c5ffda77c7cbdcb3773f"} Oct 30 19:28:12.743945 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.743916 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:28:12.754504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.754449 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podStartSLOduration=1.754402912 podStartE2EDuration="1.754402912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:12.753291854 +0000 UTC m=+666.776895755" watchObservedRunningTime="2026-10-30 19:28:12.754402912 +0000 UTC m=+666.778006810" Oct 30 19:28:12.782584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.782549 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:12.872104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:12.872072 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7"] Oct 30 19:28:12.873346 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:12.873319 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbbdf98f_0862_490f_b7f0_2357fc8782e1.slice/crio-5409afe8ec714187c9bb5e65bbdd302bdfb4eff21dc64a619ec1fc712e9888a2 WatchSource:0}: Error finding container 5409afe8ec714187c9bb5e65bbdd302bdfb4eff21dc64a619ec1fc712e9888a2: Status 404 returned error can't find the container with id 5409afe8ec714187c9bb5e65bbdd302bdfb4eff21dc64a619ec1fc712e9888a2 Oct 30 19:28:13.749342 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:13.749236 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" event={"ID":"dbbdf98f-0862-490f-b7f0-2357fc8782e1","Type":"ContainerStarted","Data":"45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d"} Oct 30 19:28:13.749342 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:13.749292 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" event={"ID":"dbbdf98f-0862-490f-b7f0-2357fc8782e1","Type":"ContainerStarted","Data":"5409afe8ec714187c9bb5e65bbdd302bdfb4eff21dc64a619ec1fc712e9888a2"} Oct 30 19:28:13.768261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:13.768210 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podStartSLOduration=1.7681960060000002 podStartE2EDuration="1.768196006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:13.766776928 +0000 UTC m=+667.790380828" watchObservedRunningTime="2026-10-30 19:28:13.768196006 +0000 UTC m=+667.791799904" Oct 30 19:28:13.784949 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:13.784919 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:13.785095 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:13.784964 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:14.312727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.312695 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85"] Oct 30 19:28:14.316202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.316183 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.353617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.353568 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85"] Oct 30 19:28:14.402682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.402645 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.402899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.402876 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.402981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.402924 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdj8d\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.402981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.402964 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.403085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.402989 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.403085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.403021 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.403085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.403055 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.403234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.403154 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.403234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.403206 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.404529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.404492 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8"] Oct 30 19:28:14.409241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.409221 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.460441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.429860 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"distinct-w5-60577a-kserve-self-signed-certs\"" Oct 30 19:28:14.460716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.460684 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8"] Oct 30 19:28:14.504941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.504902 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.504941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.504943 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.504969 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505001 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505125 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.505205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505190 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.505366 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505217 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.505451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505430 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505522 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505463 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505512 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505545 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dffgh\" (UniqueName: \"kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.505675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505574 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505572 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505658 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.505862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505716 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.505862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505744 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505762 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.505862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.505788 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdj8d\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.506199 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.506099 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.506372 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.506349 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.507968 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.507945 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.509008 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.508985 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.514207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.514178 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.514931 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.514908 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdj8d\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d\") pod \"maas-default-gateway-openshift-default-5bc884dd87-frx85\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.591655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.591526 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:14.606702 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.606671 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dffgh\" (UniqueName: \"kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.606933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.606919 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.607061 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.607050 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.607191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.607181 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.607300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.607290 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.607782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.607396 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.607782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.607482 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.607976 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.607703 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.608155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.608134 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.609696 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.609650 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.610389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.610368 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.616248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.616220 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dffgh\" (UniqueName: \"kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh\") pod \"distinct-w5-60577a-kserve-577c989554-wnjq8\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.679711 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.679682 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:14.768147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.767559 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:14.792470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.791748 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:14.792470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:14.791822 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:15.001210 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.001186 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85"] Oct 30 19:28:15.002831 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:15.002806 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480d3b55_ff1e_4e1c_a449_a9eea535eb34.slice/crio-022d19e7feeb4c5fe5feb34bcd8e79fb51c3047dea174b88ae731e74cccc9de8 WatchSource:0}: Error finding container 022d19e7feeb4c5fe5feb34bcd8e79fb51c3047dea174b88ae731e74cccc9de8: Status 404 returned error can't find the container with id 022d19e7feeb4c5fe5feb34bcd8e79fb51c3047dea174b88ae731e74cccc9de8 Oct 30 19:28:15.005186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.005151 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:15.005250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.005222 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:15.005293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.005251 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:15.098281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.098248 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8"] Oct 30 19:28:15.099326 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:15.099300 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef9d7d7_d76b_4f34_89b9_b268b43454c6.slice/crio-2830d23748f6fcb675f7f3299eb8f2353dbb0236d185e363723da04a3041607f WatchSource:0}: Error finding container 2830d23748f6fcb675f7f3299eb8f2353dbb0236d185e363723da04a3041607f: Status 404 returned error can't find the container with id 2830d23748f6fcb675f7f3299eb8f2353dbb0236d185e363723da04a3041607f Oct 30 19:28:15.775536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.774698 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" event={"ID":"def9d7d7-d76b-4f34-89b9-b268b43454c6","Type":"ContainerStarted","Data":"10e7f18b7a61f7a3bb7bff3370c30c3e9c835fa5bf159d722e3b22bfbd4fdbef"} Oct 30 19:28:15.775536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.774763 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" event={"ID":"def9d7d7-d76b-4f34-89b9-b268b43454c6","Type":"ContainerStarted","Data":"2830d23748f6fcb675f7f3299eb8f2353dbb0236d185e363723da04a3041607f"} Oct 30 19:28:15.777197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.777172 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" event={"ID":"480d3b55-ff1e-4e1c-a449-a9eea535eb34","Type":"ContainerStarted","Data":"755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a"} Oct 30 19:28:15.777258 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.777227 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" event={"ID":"480d3b55-ff1e-4e1c-a449-a9eea535eb34","Type":"ContainerStarted","Data":"022d19e7feeb4c5fe5feb34bcd8e79fb51c3047dea174b88ae731e74cccc9de8"} Oct 30 19:28:15.782652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.782629 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:15.782737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.782680 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:15.855439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.851584 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podStartSLOduration=1.8515480389999999 podStartE2EDuration="1.851548039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:15.848085632 +0000 UTC m=+669.871689534" watchObservedRunningTime="2026-10-30 19:28:15.851548039 +0000 UTC m=+669.875151938" Oct 30 19:28:15.883233 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:15.883098 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podStartSLOduration=1.883081903 podStartE2EDuration="1.883081903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:15.882936816 +0000 UTC m=+669.906540717" watchObservedRunningTime="2026-10-30 19:28:15.883081903 +0000 UTC m=+669.906685801" Oct 30 19:28:16.680939 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:16.680896 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:16.682325 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:16.682294 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:16.682492 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:16.682356 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:16.773356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:16.773327 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:16.785184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:16.784871 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:16.785184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:16.784916 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:17.694441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:17.683816 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:17.694441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:17.683891 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:17.782869 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:17.782827 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:17.783569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:17.783153 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:18.684440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:18.681956 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:18.684440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:18.682024 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:18.787448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:18.786881 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:18.787448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:18.786942 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:19.681282 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:19.681250 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:19.681551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:19.681313 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:19.782869 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:19.782833 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:28:19.783226 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:19.782903 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:28:20.680921 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:20.680874 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:20.681176 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:20.680944 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:21.066706 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:21.066667 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:21.067184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:21.067098 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:21.068138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:21.068112 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:28:21.683120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:21.683042 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:21.683325 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:21.683133 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:22.591481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.591444 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:22.604499 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.604471 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:22.680544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.680507 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:28:22.680708 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.680572 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 19:28:22.886935 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.886856 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6"] Oct 30 19:28:22.890833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.890809 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:28:22.890983 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.890884 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:22.897060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.897038 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-419d0a\"/\"distinct-2-w1-419d0a-kserve-self-signed-certs\"" Oct 30 19:28:22.904165 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.904139 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6"] Oct 30 19:28:22.983060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.982938 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:22.983060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.983049 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:22.983616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.983380 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:22.983616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.983515 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlg22\" (UniqueName: \"kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:22.983789 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.983640 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:22.983789 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:22.983731 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.086004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.085967 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlg22\" (UniqueName: \"kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.086306 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.086284 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.086493 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.086476 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.086680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.086665 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.086815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.086800 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.086949 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.086930 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.089268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.089242 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.089845 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.089815 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.090668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.090638 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.092547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.092502 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.097262 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.097240 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.101164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.100654 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlg22\" (UniqueName: \"kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22\") pod \"distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.209352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.209259 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:23.506889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.506856 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6"] Oct 30 19:28:23.507855 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:23.507817 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c9fc379_4173_49c9_aaeb_e85ce3755b0a.slice/crio-a6d93da6129f720decfcb4bd7cc83f496f71a55e37d7305c644da3896ea2130e WatchSource:0}: Error finding container a6d93da6129f720decfcb4bd7cc83f496f71a55e37d7305c644da3896ea2130e: Status 404 returned error can't find the container with id a6d93da6129f720decfcb4bd7cc83f496f71a55e37d7305c644da3896ea2130e Oct 30 19:28:23.897333 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.897297 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" event={"ID":"4c9fc379-4173-49c9-aaeb-e85ce3755b0a","Type":"ContainerStarted","Data":"8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb"} Oct 30 19:28:23.897802 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.897349 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" event={"ID":"4c9fc379-4173-49c9-aaeb-e85ce3755b0a","Type":"ContainerStarted","Data":"a6d93da6129f720decfcb4bd7cc83f496f71a55e37d7305c644da3896ea2130e"} Oct 30 19:28:23.917235 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.917177 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podStartSLOduration=1.917157155 podStartE2EDuration="1.917157155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:23.915601816 +0000 UTC m=+677.939205716" watchObservedRunningTime="2026-10-30 19:28:23.917157155 +0000 UTC m=+677.940761054" Oct 30 19:28:23.986313 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.986041 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:23.986578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.986554 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:23.987794 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:23.987767 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:28:24.210278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:24.210205 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:24.377968 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:28:24.377932 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:28:24.768336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:24.768305 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:24.781011 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:24.780983 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:24.914525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:24.914497 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:28:28.640789 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.640755 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz"] Oct 30 19:28:28.645384 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.645358 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.653838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.653810 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz"] Oct 30 19:28:28.656255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.656231 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"default-dockercfg-x2w78\"" Oct 30 19:28:28.656381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.656254 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"openshift-service-ca.crt\"" Oct 30 19:28:28.656381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.656257 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"kube-root-ca.crt\"" Oct 30 19:28:28.656381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.656300 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ce69f3\"/\"distinct-2-w3-ce69f3-kserve-self-signed-certs\"" Oct 30 19:28:28.760850 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.760693 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.761022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.760901 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.761022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.760964 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7hxg\" (UniqueName: \"kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.761022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.761004 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.761156 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.761037 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.761156 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.761074 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.863523 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.863476 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.863728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.863633 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.863728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.863682 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7hxg\" (UniqueName: \"kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.863728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.863720 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.863927 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.863748 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.863927 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.863792 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.864951 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.864919 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.865289 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.865265 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.865538 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.865511 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.867669 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.867647 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.867822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.867661 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.879572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.879547 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7hxg\" (UniqueName: \"kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg\") pod \"distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:28.962127 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:28.962039 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:29.257017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:29.256983 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz"] Oct 30 19:28:29.259033 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:29.259006 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c64d263_5c5a_4c5b_891c_4607e6e850a3.slice/crio-e013278f92e446d98cb54c1b304dac4b2952b634ccd96a2941704f66a15d0481 WatchSource:0}: Error finding container e013278f92e446d98cb54c1b304dac4b2952b634ccd96a2941704f66a15d0481: Status 404 returned error can't find the container with id e013278f92e446d98cb54c1b304dac4b2952b634ccd96a2941704f66a15d0481 Oct 30 19:28:29.939271 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:29.939193 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" event={"ID":"3c64d263-5c5a-4c5b-891c-4607e6e850a3","Type":"ContainerStarted","Data":"235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3"} Oct 30 19:28:29.939271 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:29.939262 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" event={"ID":"3c64d263-5c5a-4c5b-891c-4607e6e850a3","Type":"ContainerStarted","Data":"e013278f92e446d98cb54c1b304dac4b2952b634ccd96a2941704f66a15d0481"} Oct 30 19:28:29.958124 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:29.958074 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podStartSLOduration=1.958061443 podStartE2EDuration="1.958061443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:29.957091811 +0000 UTC m=+683.980695712" watchObservedRunningTime="2026-10-30 19:28:29.958061443 +0000 UTC m=+683.981665342" Oct 30 19:28:29.962587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:29.962567 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:33.211128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:33.209863 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:33.316717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:33.316689 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:34.069285 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.068955 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:28:34.165064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.165034 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd"] Oct 30 19:28:34.171397 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.171371 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd"] Oct 30 19:28:34.171727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.171709 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.176367 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.176344 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"kube-root-ca.crt\"" Oct 30 19:28:34.177287 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.177269 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"default-dockercfg-qkj8m\"" Oct 30 19:28:34.177544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.177526 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"distinct-w4-ad96a4-kserve-self-signed-certs\"" Oct 30 19:28:34.177729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.177713 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"openshift-service-ca.crt\"" Oct 30 19:28:34.277445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.277102 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.277445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.277154 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fz4\" (UniqueName: \"kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.277445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.277235 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.277445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.277309 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.277445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.277372 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.277445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.277463 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.378956 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379086 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379113 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fz4\" (UniqueName: \"kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379176 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379227 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379281 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379389 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379604 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.379687 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.379655 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.381575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.381549 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.381932 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.381909 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.387715 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.387690 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fz4\" (UniqueName: \"kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4\") pod \"distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.498119 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.498082 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:34.785621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.785588 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd"] Oct 30 19:28:34.786279 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:34.786242 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda06946fa_d410_46a0_a049_442b7d2f7949.slice/crio-36ed3c641f4bb1b85b994418ccc2d7abee3ac7afe76b095eb2ebc09c534e74b7 WatchSource:0}: Error finding container 36ed3c641f4bb1b85b994418ccc2d7abee3ac7afe76b095eb2ebc09c534e74b7: Status 404 returned error can't find the container with id 36ed3c641f4bb1b85b994418ccc2d7abee3ac7afe76b095eb2ebc09c534e74b7 Oct 30 19:28:34.990253 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.990211 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" event={"ID":"a06946fa-d410-46a0-a049-442b7d2f7949","Type":"ContainerStarted","Data":"9074dec1ef1f9a1d715011bcdc1fed08d04fc2c237fa530d55120fdcf901c862"} Oct 30 19:28:34.990482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:34.990280 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" event={"ID":"a06946fa-d410-46a0-a049-442b7d2f7949","Type":"ContainerStarted","Data":"36ed3c641f4bb1b85b994418ccc2d7abee3ac7afe76b095eb2ebc09c534e74b7"} Oct 30 19:28:35.010978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:35.010912 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podStartSLOduration=1.010881053 podStartE2EDuration="1.010881053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:35.00897557 +0000 UTC m=+689.032579470" watchObservedRunningTime="2026-10-30 19:28:35.010881053 +0000 UTC m=+689.034484953" Oct 30 19:28:35.498472 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:35.498410 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:38.962904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:38.962860 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:38.976481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:38.976452 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:39.041588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:39.041560 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:28:41.386999 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.386962 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk"] Oct 30 19:28:41.391628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.391602 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.420348 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.420318 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk"] Oct 30 19:28:41.454741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.454713 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.454858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.454743 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.454858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.454769 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.454953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.454928 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.454990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.454978 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.455071 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.455056 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.455107 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.455083 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.455243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.455228 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.455280 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.455257 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52mqj\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556245 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556320 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556339 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556450 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556488 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52mqj\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556541 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556564 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556598 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556674 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556720 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556779 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.556844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.556819 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.557061 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.557041 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.557308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.557292 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.559138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.559111 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.559554 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.559533 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.586691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.586662 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.593767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.593746 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52mqj\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.707100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.707018 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:41.995579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.995553 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk"] Oct 30 19:28:41.996612 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:41.996570 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2ff7d8_7b6b_402e_b14d_692343dd1f22.slice/crio-daa1bca035fdb0e79b17d2834427be983c00819d885ebb84c17b6707ef8e658c WatchSource:0}: Error finding container daa1bca035fdb0e79b17d2834427be983c00819d885ebb84c17b6707ef8e658c: Status 404 returned error can't find the container with id daa1bca035fdb0e79b17d2834427be983c00819d885ebb84c17b6707ef8e658c Oct 30 19:28:41.999235 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.999201 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:41.999305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.999266 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:41.999305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:41.999296 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:42.048309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:42.048279 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" event={"ID":"ff2ff7d8-7b6b-402e-b14d-692343dd1f22","Type":"ContainerStarted","Data":"daa1bca035fdb0e79b17d2834427be983c00819d885ebb84c17b6707ef8e658c"} Oct 30 19:28:43.056645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.056610 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" event={"ID":"ff2ff7d8-7b6b-402e-b14d-692343dd1f22","Type":"ContainerStarted","Data":"6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e"} Oct 30 19:28:43.083322 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.083269 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podStartSLOduration=2.083255244 podStartE2EDuration="2.083255244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:43.079398656 +0000 UTC m=+697.103002555" watchObservedRunningTime="2026-10-30 19:28:43.083255244 +0000 UTC m=+697.106859143" Oct 30 19:28:43.196029 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.195999 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp"] Oct 30 19:28:43.200174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.200151 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.209787 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.209312 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp"] Oct 30 19:28:43.210448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.210396 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-60577a\"/\"distinct-2-w5-60577a-kserve-self-signed-certs\"" Oct 30 19:28:43.279454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.279394 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.279641 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.279598 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.279709 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.279674 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.279833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.279809 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mts6\" (UniqueName: \"kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.279919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.279902 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.279957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.279927 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.383186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.383082 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.383941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.383915 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.384126 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.384106 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mts6\" (UniqueName: \"kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.384260 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.384245 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.384357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.384344 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.384542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.384524 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.384754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.384736 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.385200 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.385181 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.386160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.386140 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.399743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.399715 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.404041 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.404009 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.409769 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.409227 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mts6\" (UniqueName: \"kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6\") pod \"distinct-2-w5-60577a-kserve-cc79b8569-n5fmp\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.553785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.553748 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:43.707483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.707385 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:43.708643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.708617 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:28:43.708781 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.708671 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 19:28:43.847067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:43.847034 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp"] Oct 30 19:28:43.847689 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:43.847665 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e9bd48d_c401_4276_95a0_96c1fccff988.slice/crio-a40670cf2b1f6d3a6dd1986e4ec16fefa900726a7f7faadba9c6b0095dde3aa6 WatchSource:0}: Error finding container a40670cf2b1f6d3a6dd1986e4ec16fefa900726a7f7faadba9c6b0095dde3aa6: Status 404 returned error can't find the container with id a40670cf2b1f6d3a6dd1986e4ec16fefa900726a7f7faadba9c6b0095dde3aa6 Oct 30 19:28:44.064978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.064936 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" event={"ID":"7e9bd48d-c401-4276-95a0-96c1fccff988","Type":"ContainerStarted","Data":"3c3be8b1b164d2a35a297344f9137f1555529c131353aab3ad5ca377863ee1e1"} Oct 30 19:28:44.065387 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.064990 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" event={"ID":"7e9bd48d-c401-4276-95a0-96c1fccff988","Type":"ContainerStarted","Data":"a40670cf2b1f6d3a6dd1986e4ec16fefa900726a7f7faadba9c6b0095dde3aa6"} Oct 30 19:28:44.082882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.082835 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podStartSLOduration=1.082821968 podStartE2EDuration="1.082821968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:44.081824584 +0000 UTC m=+698.105428485" watchObservedRunningTime="2026-10-30 19:28:44.082821968 +0000 UTC m=+698.106425867" Oct 30 19:28:44.499034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.498952 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:44.511692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.511667 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:44.553878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.553831 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:44.708032 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.707999 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:28:44.708193 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:44.708045 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 19:28:45.083028 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:45.083000 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:28:45.707542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:45.707509 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:28:45.707698 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:45.707562 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 19:28:46.708388 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:46.708353 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:28:46.708783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:46.708407 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 19:28:47.708129 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:47.708092 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 19:28:47.708306 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:47.708155 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 19:28:48.980864 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:48.980834 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:48.981363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:48.981317 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:48.982082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:48.982061 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:28:53.554940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:53.554902 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:53.567676 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:53.567648 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:54.134451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.134404 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:28:54.379446 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:28:54.379376 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:28:54.658531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.658492 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb"] Oct 30 19:28:54.663266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.663182 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.691372 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.691342 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb"] Oct 30 19:28:54.765956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.765918 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b623ca\"/\"distinct-2-w2-b623ca-kserve-self-signed-certs\"" Oct 30 19:28:54.852150 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.852106 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.852551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.852526 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.852682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.852667 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.852869 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.852851 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w467\" (UniqueName: \"kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.852990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.852974 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.853093 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.853079 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.954433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.954302 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w467\" (UniqueName: \"kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.954760 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.954738 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.955541 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.955390 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.955895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.955813 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.956016 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.955927 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.959433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.956379 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.959433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.956527 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.959433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.956758 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.964205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.961259 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.964205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.962004 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.964205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.962740 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.968933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.968910 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w467\" (UniqueName: \"kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467\") pod \"distinct-2-w2-b623ca-kserve-995469d67-gl6tb\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:54.980449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:54.977042 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:55.343185 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:55.341919 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb"] Oct 30 19:28:55.360465 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:28:55.351080 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ef7861_ba7f_4358_8f3b_ffeaca572ee7.slice/crio-a4a9816b2a9a7a55c3a2e23cb5c344333d91da415884b496b38383540a3279b0 WatchSource:0}: Error finding container a4a9816b2a9a7a55c3a2e23cb5c344333d91da415884b496b38383540a3279b0: Status 404 returned error can't find the container with id a4a9816b2a9a7a55c3a2e23cb5c344333d91da415884b496b38383540a3279b0 Oct 30 19:28:56.238439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.238031 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" event={"ID":"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7","Type":"ContainerStarted","Data":"03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5"} Oct 30 19:28:56.238439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.238088 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" event={"ID":"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7","Type":"ContainerStarted","Data":"a4a9816b2a9a7a55c3a2e23cb5c344333d91da415884b496b38383540a3279b0"} Oct 30 19:28:56.334268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.334209 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podStartSLOduration=2.334190203 podStartE2EDuration="2.334190203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:56.333071165 +0000 UTC m=+710.356675066" watchObservedRunningTime="2026-10-30 19:28:56.334190203 +0000 UTC m=+710.357794104" Oct 30 19:28:56.818440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.818136 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg"] Oct 30 19:28:56.832527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.823031 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.854439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.839483 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7"] Oct 30 19:28:56.854439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.849122 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.875439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.867047 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg"] Oct 30 19:28:56.888694 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.888642 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911175 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911225 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911269 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911361 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpn4j\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911401 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911459 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911485 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911524 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911569 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911600 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911637 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911693 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911741 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911775 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911852 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911884 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911944 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bxh\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:56.919566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.911968 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:56.954819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.938782 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.990930 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:56.992857 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.024153 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.024201 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027475 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bxh\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027534 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027586 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027614 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027652 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027758 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpn4j\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027805 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027847 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027873 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027911 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027960 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.027990 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.028027 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.028089 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.028143 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.028175 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.028669 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.028915 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.029142 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.043388 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.044376 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7"] Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.044815 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.045940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.045349 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.048137 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.048369 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.048610 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.060157 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.108239 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.108866 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.109150 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.120955 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.121771 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bxh\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.141525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.127777 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:28:57.163111 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.162916 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.181720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.169172 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.181720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.172741 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.185633 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.185584 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpn4j\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j\") pod \"e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.286461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.285669 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:57.359998 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.356577 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:57.763892 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.763459 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:57.763892 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.763550 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:57.763892 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.763620 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:57.764159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.763998 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7"] Oct 30 19:28:57.764790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.764749 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:57.764904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.764817 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:57.764904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.764860 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:28:57.777294 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:57.777259 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg"] Oct 30 19:28:58.270335 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.270296 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" event={"ID":"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740","Type":"ContainerStarted","Data":"4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236"} Oct 30 19:28:58.270562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.270367 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" event={"ID":"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740","Type":"ContainerStarted","Data":"51cf24e014213355bbe5c239808798308f811246b367137c8fcc362ffb244c3c"} Oct 30 19:28:58.276284 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.276253 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" event={"ID":"4ed52396-d268-4772-bf2c-b6cc27a0ed09","Type":"ContainerStarted","Data":"3b5a40601d947ddb8efa4f9bfff4c2dcd786f932677c08520475743eac700dda"} Oct 30 19:28:58.276394 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.276309 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" event={"ID":"4ed52396-d268-4772-bf2c-b6cc27a0ed09","Type":"ContainerStarted","Data":"9f10b6d1ea2995786fe5d383e4337e1b1d29d0c858793529f359d0a3934ca006"} Oct 30 19:28:58.288063 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.288033 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:28:58.288485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.288135 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:28:58.288485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.288176 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:28:58.299447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.299367 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podStartSLOduration=2.299354524 podStartE2EDuration="2.299354524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:58.295653434 +0000 UTC m=+712.319257336" watchObservedRunningTime="2026-10-30 19:28:58.299354524 +0000 UTC m=+712.322958423" Oct 30 19:28:58.320349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.320301 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podStartSLOduration=2.320287372 podStartE2EDuration="2.320287372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:58.318344993 +0000 UTC m=+712.341948893" watchObservedRunningTime="2026-10-30 19:28:58.320287372 +0000 UTC m=+712.343891271" Oct 30 19:28:58.357968 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.357938 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:28:58.359159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.359135 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:28:58.359275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:58.359189 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:28:59.287053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:59.287017 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:28:59.287222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:59.287083 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:28:59.382853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:59.382714 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:28:59.383347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:28:59.382884 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:29:00.283439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.283287 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227"] Oct 30 19:29:00.288940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.288909 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.289173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.289145 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:29:00.293255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.291739 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.311773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.310757 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad96a4\"/\"distinct-2-w4-ad96a4-kserve-self-signed-certs\"" Oct 30 19:29:00.316721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.316673 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.317015 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.316992 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.317191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.317171 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.317446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.317399 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7ds\" (UniqueName: \"kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.317644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.317626 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.317801 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.317784 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.421441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.418005 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227"] Oct 30 19:29:00.422035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.422007 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.422228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.422201 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:29:00.516449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.515708 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7ds\" (UniqueName: \"kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.516449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.515792 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.516449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.515856 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.516449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.515928 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.516449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.516037 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.516449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.516110 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.516919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.516687 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.522437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.520106 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.522437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.520360 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.554304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.550708 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7ds\" (UniqueName: \"kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.577874 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.577837 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.578685 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.578661 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs\") pod \"distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:00.761675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:00.761644 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:01.144441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.143927 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227"] Oct 30 19:29:01.286522 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.286498 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.286620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.286536 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:29:01.366092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.365799 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.366092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.365861 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:29:01.487793 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.487700 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" event={"ID":"a258aebe-81aa-4b03-a7ff-0aca873e5cc6","Type":"ContainerStarted","Data":"0863844bf6954cb523e0d28dab12270fab477ad261aa4fb2cf123bd95753b36a"} Oct 30 19:29:01.488332 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.488307 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" event={"ID":"a258aebe-81aa-4b03-a7ff-0aca873e5cc6","Type":"ContainerStarted","Data":"66bad68f97e83e46f25cf306d6c73f06231bdaafe7608e79110c18711efdefc0"} Oct 30 19:29:01.508513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.508468 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podStartSLOduration=1.508456082 podStartE2EDuration="1.508456082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:01.507907886 +0000 UTC m=+715.531511785" watchObservedRunningTime="2026-10-30 19:29:01.508456082 +0000 UTC m=+715.532059980" Oct 30 19:29:01.767496 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:01.766391 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:02.288450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:02.287209 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.288450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:02.287268 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:29:02.362440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:02.361874 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.362440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:02.361938 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:29:03.286823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:03.286789 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.287218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:03.286845 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:29:03.363476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:03.360823 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.363476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:03.360902 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:29:04.286537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:04.286497 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.286779 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:04.286588 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 19:29:04.357957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:04.357921 6511 patch_prober.go:29] interesting pod/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.358328 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:04.357986 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 19:29:04.977491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:04.977445 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:29:04.998446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:04.998389 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:29:05.529439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.528582 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:29:05.529909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.529646 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:29:05.533408 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.533380 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:29:05.535446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.534574 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:29:05.535575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.535538 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:29:05.536512 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.536490 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:29:05.542750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:05.542727 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:29:10.762284 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:10.762247 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:10.775400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:10.775377 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:11.579487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.579444 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:11.718300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.718263 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp"] Oct 30 19:29:11.728255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.728225 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.745792 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.745761 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp"] Oct 30 19:29:11.752607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.752050 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79"] Oct 30 19:29:11.755971 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.755951 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.769504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.769027 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79"] Oct 30 19:29:11.813701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.813670 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.813890 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.813708 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.813890 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.813737 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.813895 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xtqv\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.814017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.813949 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.813976 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814015 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814238 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814065 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814238 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814088 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.814238 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814130 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbs5m\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814276 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.814400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814305 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814347 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.814400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814385 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814738 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814448 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.814738 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814468 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.814738 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814506 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.814738 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.814546 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.915261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915171 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.915261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915236 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915284 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xtqv\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.915543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915310 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915333 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915361 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915478 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915866 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915866 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.915956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915798 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.915956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.915924 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbs5m\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.916252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916231 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.916314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916266 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.916466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916440 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.916657 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916618 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.916752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916677 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.916752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916631 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.916752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916711 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.916919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916763 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.916919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916784 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.916919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916819 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.916919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916877 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.917122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.917023 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.917122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.916521 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.917252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.917229 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.917315 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.917295 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.917372 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.917300 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.917603 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.917582 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.918191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.918167 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.918530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.918501 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.919401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.919384 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.919992 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.919974 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.924850 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.924816 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.925078 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.925060 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xtqv\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:11.925369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.925335 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbs5m\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:11.925716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:11.925695 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token\") pod \"e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:12.057810 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.057770 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:12.079948 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.079916 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:12.386531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.386504 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp"] Oct 30 19:29:12.387331 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:29:12.387304 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f18a422_a162_471f_973c_7f3c532c977a.slice/crio-7fdeca053451ede21660503185bcd99e5252826523216059799d52ae8d19aa9b WatchSource:0}: Error finding container 7fdeca053451ede21660503185bcd99e5252826523216059799d52ae8d19aa9b: Status 404 returned error can't find the container with id 7fdeca053451ede21660503185bcd99e5252826523216059799d52ae8d19aa9b Oct 30 19:29:12.389862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.389828 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:12.389959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.389904 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:12.389959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.389935 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:12.405457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.405426 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79"] Oct 30 19:29:12.411302 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:29:12.411275 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb67505d_0f56_4867_9f46_481643eba36b.slice/crio-196fe3c5700160d490580ae6228aa1f85601d13de355883fad9f1911e07b8e99 WatchSource:0}: Error finding container 196fe3c5700160d490580ae6228aa1f85601d13de355883fad9f1911e07b8e99: Status 404 returned error can't find the container with id 196fe3c5700160d490580ae6228aa1f85601d13de355883fad9f1911e07b8e99 Oct 30 19:29:12.413906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.413871 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:12.413987 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.413968 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:12.414035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.414021 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:12.577936 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.577606 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" event={"ID":"4f18a422-a162-471f-973c-7f3c532c977a","Type":"ContainerStarted","Data":"3fab75fe11b7673bef46eac2b5c79fe48c13b9fa29b47330d5674ac3485df926"} Oct 30 19:29:12.577936 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.577670 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" event={"ID":"4f18a422-a162-471f-973c-7f3c532c977a","Type":"ContainerStarted","Data":"7fdeca053451ede21660503185bcd99e5252826523216059799d52ae8d19aa9b"} Oct 30 19:29:12.587199 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.587005 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" event={"ID":"db67505d-0f56-4867-9f46-481643eba36b","Type":"ContainerStarted","Data":"9723fc2ff6fa1e7b08971e9f7ffb7f442e3cc832b80f697b89b878edaa25465b"} Oct 30 19:29:12.587199 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.587062 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" event={"ID":"db67505d-0f56-4867-9f46-481643eba36b","Type":"ContainerStarted","Data":"196fe3c5700160d490580ae6228aa1f85601d13de355883fad9f1911e07b8e99"} Oct 30 19:29:12.613859 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.613809 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podStartSLOduration=1.613790411 podStartE2EDuration="1.613790411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:12.60853015 +0000 UTC m=+726.632134049" watchObservedRunningTime="2026-10-30 19:29:12.613790411 +0000 UTC m=+726.637394311" Oct 30 19:29:12.642326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:12.642264 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podStartSLOduration=1.642246093 podStartE2EDuration="1.642246093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:12.638151024 +0000 UTC m=+726.661754933" watchObservedRunningTime="2026-10-30 19:29:12.642246093 +0000 UTC m=+726.665849994" Oct 30 19:29:13.067818 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:13.067784 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:13.072352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:13.072318 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:13.072550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:13.072375 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:13.080303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:13.080273 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:13.083020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:13.082997 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:13.083142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:13.083042 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:14.069816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:14.069777 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:14.070816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:14.070387 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:14.080566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:14.080542 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:14.080692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:14.080592 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:15.062811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:15.062772 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:15.063104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:15.063072 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:15.080568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:15.080535 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:15.081132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:15.081104 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:16.058780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:16.058741 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:16.059049 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:16.059019 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:16.081402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:16.081370 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:16.081946 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:16.081921 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:17.058440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.058200 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:17.058440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.058262 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:17.081714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.081683 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:17.082163 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.081735 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:17.610958 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.602326 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26"] Oct 30 19:29:17.616458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.613942 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26"] Oct 30 19:29:17.616458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.614050 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.640288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.640000 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"openshift-service-ca.crt\"" Oct 30 19:29:17.640288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.640105 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"default-dockercfg-95md7\"" Oct 30 19:29:17.640288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.640189 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"kube-root-ca.crt\"" Oct 30 19:29:17.640288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.640238 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ae63bf\"/\"distinct-2-w0-ae63bf-kserve-self-signed-certs\"" Oct 30 19:29:17.772443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.771213 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.772443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.771272 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfr6c\" (UniqueName: \"kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.772443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.771335 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.772443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.771456 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.772443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.771526 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.772443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.771552 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.877500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.877201 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.877500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.877296 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.877500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.877323 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.877500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.877465 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.877834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.877501 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfr6c\" (UniqueName: \"kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.877834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.877558 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.904440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.902561 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.904440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.902863 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.912369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.909574 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.917453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.913797 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.919770 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.919738 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:17.926482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:17.926431 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfr6c\" (UniqueName: \"kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c\") pod \"distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:18.004143 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.004112 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:18.058948 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.058889 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:18.059248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.059198 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:18.087052 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.087016 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:18.087620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.087590 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:18.473858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.473785 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26"] Oct 30 19:29:18.743996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.743906 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" event={"ID":"df8574bd-d1a6-4b82-8004-5a719d825265","Type":"ContainerStarted","Data":"e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0"} Oct 30 19:29:18.743996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.743958 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" event={"ID":"df8574bd-d1a6-4b82-8004-5a719d825265","Type":"ContainerStarted","Data":"904feda20aa432dd65be51af08c4c5c643168ef72f1528a2a0549998816cf707"} Oct 30 19:29:18.762563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:18.762517 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podStartSLOduration=1.762504296 podStartE2EDuration="1.762504296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:18.761704193 +0000 UTC m=+732.785308093" watchObservedRunningTime="2026-10-30 19:29:18.762504296 +0000 UTC m=+732.786108194" Oct 30 19:29:19.005448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:19.005354 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:19.058340 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:19.058304 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 19:29:19.058545 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:19.058361 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 19:29:19.080898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:19.080863 6511 patch_prober.go:29] interesting pod/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 19:29:19.081076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:19.080924 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 19:29:20.062907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:20.062875 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:20.063309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:20.063160 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:20.063971 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:20.063953 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:29:20.084916 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:20.084894 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:20.085191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:20.085167 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:20.085871 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:20.085854 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:29:24.379764 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:29:24.379724 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:29:27.293071 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.293032 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps"] Oct 30 19:29:27.322681 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.322646 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps"] Oct 30 19:29:27.323018 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.322998 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.506609 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskcx\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.506675 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.506749 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.506827 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.506887 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.506918 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.507015 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.507043 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.516466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.507070 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.628055 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.628000 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskcx\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.628359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.628332 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.643642 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.628944 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.645113 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.645076 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.660146 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.660232 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.660262 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.660371 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.660395 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.660441 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.665925 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.666180 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.667098 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.679514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.668458 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.720100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.705742 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.743786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.743749 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.752438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.744741 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskcx\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.759530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.758124 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.866813 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.866778 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:27.878883 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:27.878845 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:28.010507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.009787 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:28.455690 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.455660 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:28.693304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.691708 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:28.693304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.691813 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:28.693304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.691873 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:28.708498 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.702900 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps"] Oct 30 19:29:28.876592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.876559 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" event={"ID":"5744deea-ed79-49ad-a2e1-7472390c0a1b","Type":"ContainerStarted","Data":"5e78811d47c58637e9a90001aca721f68cdb902ef370784d8e24cfe7274de23b"} Oct 30 19:29:28.876795 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.876614 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" event={"ID":"5744deea-ed79-49ad-a2e1-7472390c0a1b","Type":"ContainerStarted","Data":"93f7a7bb71ae6a9640e266862eca9dd9f8f456b66b2e5708700879bf2ff429ef"} Oct 30 19:29:28.898910 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.898879 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:28.909218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:28.909157 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podStartSLOduration=1.909138187 podStartE2EDuration="1.909138187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:28.905792568 +0000 UTC m=+742.929396468" watchObservedRunningTime="2026-10-30 19:29:28.909138187 +0000 UTC m=+742.932742090" Oct 30 19:29:29.867986 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:29.867950 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:29.870996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:29.869546 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:29:29.871156 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:29.871036 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:29:30.868173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:30.868132 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:29:30.868716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:30.868199 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:29:31.868430 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:31.868382 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:29:31.868794 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:31.868477 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:29:32.867828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:32.867796 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:29:32.868009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:32.867854 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:29:33.868394 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:33.868364 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:29:33.868869 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:33.868838 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:29:34.868105 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:34.868041 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:29:34.868274 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:34.868126 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:29:36.338738 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:36.338701 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:36.339441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:36.339164 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:36.340036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:36.340013 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:29:41.576506 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.576473 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm"] Oct 30 19:29:41.582959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.582931 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.592387 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.592359 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm"] Oct 30 19:29:41.661214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.661178 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8"] Oct 30 19:29:41.667696 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.667671 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.682724 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.682700 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8"] Oct 30 19:29:41.751398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751368 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751403 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751456 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751492 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751603 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751623 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751638 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5ql\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751813 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751790 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.751913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.751899 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.853279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853201 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.853279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853256 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.853560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853331 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.853560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853370 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.853560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853396 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.853560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853481 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.853560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853518 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.853560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853556 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.853866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853596 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4n5\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.853866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853634 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.853866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853757 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.853866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.853783 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.854085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854004 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854038 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854178 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854113 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854178 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854146 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854175 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5ql\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854405 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.854501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854483 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.854645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854623 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854634 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854758 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854744 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.854971 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.854951 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.856095 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.856076 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.856354 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.856333 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.872521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.872485 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.874803 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.874777 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5ql\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql\") pod \"e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.910900 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.910864 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:41.956134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956095 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956160 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956206 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4n5\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956241 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956288 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956311 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956664 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956371 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956664 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956407 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.956664 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.956608 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.957163 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.957132 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.957369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.957334 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.957568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.957387 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.957857 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.957836 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.958138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.958050 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.960487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.960466 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.960654 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.960637 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.966028 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.966004 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4n5\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.966287 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.966268 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token\") pod \"e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:41.996681 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:41.990061 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:42.164234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.164161 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z"] Oct 30 19:29:42.169109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.169091 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.186752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.186729 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z"] Oct 30 19:29:42.205018 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.204994 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm"] Oct 30 19:29:42.205740 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:29:42.205718 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b0be6f_1786_4ef6_890b_217f626c17f2.slice/crio-07efa832883f84169eec18ee3c763f7dd8ff5809da40394cd8196c19ad8dda5e WatchSource:0}: Error finding container 07efa832883f84169eec18ee3c763f7dd8ff5809da40394cd8196c19ad8dda5e: Status 404 returned error can't find the container with id 07efa832883f84169eec18ee3c763f7dd8ff5809da40394cd8196c19ad8dda5e Oct 30 19:29:42.208936 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.208903 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.209044 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.208970 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.209044 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.209001 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.286925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.286900 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8"] Oct 30 19:29:42.287006 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:29:42.286960 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ec9afe_6e5d_4fb3_8431_d135790c9919.slice/crio-2fc5c4f1107462d551394906321a16c50a7e5dd870c7faaa73879e90dc0c7bb1 WatchSource:0}: Error finding container 2fc5c4f1107462d551394906321a16c50a7e5dd870c7faaa73879e90dc0c7bb1: Status 404 returned error can't find the container with id 2fc5c4f1107462d551394906321a16c50a7e5dd870c7faaa73879e90dc0c7bb1 Oct 30 19:29:42.289828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.289786 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.289936 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.289915 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.289985 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.289968 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.362761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.362730 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.362878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.362791 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.362953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.362930 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.363090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.363063 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mtv\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.363153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.363114 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.363299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.363274 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.363450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.363391 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.363526 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.363494 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.363576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.363535 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465386 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465487 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465735 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465535 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465735 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465650 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mtv\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465735 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465684 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465780 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465839 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.465909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465875 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.466079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465908 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.466079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465934 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.466079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.465957 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.466539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.466515 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.466701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.466616 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.466829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.466805 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.468279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.468258 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.469684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.469655 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.476625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.476594 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mtv\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.476756 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.476736 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.484708 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.484689 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:42.800870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.800837 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z"] Oct 30 19:29:42.802182 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:29:42.802155 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c12a0a_2a8b_4132_ac2e_11e9a18beccd.slice/crio-ef35724fe37c838c9fbc7966b905d55342c7b45db8110a08816116fc237e44fc WatchSource:0}: Error finding container ef35724fe37c838c9fbc7966b905d55342c7b45db8110a08816116fc237e44fc: Status 404 returned error can't find the container with id ef35724fe37c838c9fbc7966b905d55342c7b45db8110a08816116fc237e44fc Oct 30 19:29:42.806014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.805978 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.806132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.806045 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:42.806132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:42.806077 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:29:43.013925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.013652 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" event={"ID":"f5b0be6f-1786-4ef6-890b-217f626c17f2","Type":"ContainerStarted","Data":"9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5"} Oct 30 19:29:43.013925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.013711 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" event={"ID":"f5b0be6f-1786-4ef6-890b-217f626c17f2","Type":"ContainerStarted","Data":"07efa832883f84169eec18ee3c763f7dd8ff5809da40394cd8196c19ad8dda5e"} Oct 30 19:29:43.015627 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.015546 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" event={"ID":"84c12a0a-2a8b-4132-ac2e-11e9a18beccd","Type":"ContainerStarted","Data":"18e8acf39724d0239cadb25865a435a69bfdf0742a49d2446bbef3d89f0d5f8b"} Oct 30 19:29:43.015627 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.015601 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" event={"ID":"84c12a0a-2a8b-4132-ac2e-11e9a18beccd","Type":"ContainerStarted","Data":"ef35724fe37c838c9fbc7966b905d55342c7b45db8110a08816116fc237e44fc"} Oct 30 19:29:43.042438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.017344 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" event={"ID":"42ec9afe-6e5d-4fb3-8431-d135790c9919","Type":"ContainerStarted","Data":"e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82"} Oct 30 19:29:43.042438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.017374 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" event={"ID":"42ec9afe-6e5d-4fb3-8431-d135790c9919","Type":"ContainerStarted","Data":"2fc5c4f1107462d551394906321a16c50a7e5dd870c7faaa73879e90dc0c7bb1"} Oct 30 19:29:43.108457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.108145 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podStartSLOduration=2.108130102 podStartE2EDuration="2.108130102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:43.083509905 +0000 UTC m=+757.107113808" watchObservedRunningTime="2026-10-30 19:29:43.108130102 +0000 UTC m=+757.131734001" Oct 30 19:29:43.137445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.133806 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podStartSLOduration=1.133785246 podStartE2EDuration="1.133785246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:43.107523954 +0000 UTC m=+757.131127856" watchObservedRunningTime="2026-10-30 19:29:43.133785246 +0000 UTC m=+757.157389145" Oct 30 19:29:43.184442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.183797 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podStartSLOduration=2.183778906 podStartE2EDuration="2.183778906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:43.132689821 +0000 UTC m=+757.156293722" watchObservedRunningTime="2026-10-30 19:29:43.183778906 +0000 UTC m=+757.207382850" Oct 30 19:29:43.486197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.485581 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:43.487082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.487048 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:43.487213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.487110 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:43.912441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.912187 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:43.914836 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.914799 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:29:43.914982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:43.914860 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:29:44.003636 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.003595 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:44.009013 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.007552 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:29:44.009013 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.007608 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:29:44.488441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.485832 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:44.488441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.485892 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:44.917454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.911755 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:29:44.917454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.911815 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:29:44.995613 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.995578 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:29:44.995909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:44.995880 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:29:45.492440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:45.489832 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:45.492440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:45.489895 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:45.911699 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:45.911653 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:29:45.911894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:45.911726 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:29:45.997451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:45.996880 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:29:45.997451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:45.996945 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:29:46.497487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:46.492843 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:46.497487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:46.492901 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:46.911539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:46.911498 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:29:46.911741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:46.911555 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:29:46.992735 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:46.992701 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:29:46.992975 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:46.992950 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:29:47.516439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:47.514689 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:47.516439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:47.514749 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:47.912943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:47.912900 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:29:47.913208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:47.913176 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:29:47.998438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:47.997651 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:29:47.998438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:47.997722 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:29:48.486015 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:48.485960 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.486220 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:48.486028 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:48.911668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:48.911636 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.911668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:48.911689 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:29:48.992973 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:48.992941 6511 patch_prober.go:29] interesting pod/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:29:48.993156 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:48.992997 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 19:29:49.485200 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:49.485157 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:29:49.485398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:49.485243 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 19:29:50.134449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.128240 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:50.134449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.129137 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:50.138094 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.138072 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:29:50.298344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.298312 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:50.300454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.300426 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:50.301102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.301084 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:29:50.830704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.830667 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:50.831184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.831160 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:50.831998 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:50.831976 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:29:54.378393 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:29:54.378360 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:29:57.523346 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.523308 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6"] Oct 30 19:29:57.884448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.869112 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6"] Oct 30 19:29:57.884448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.869257 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:57.892334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.891012 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:57.926441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.926379 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:57.927615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.927486 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:57.927615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.927593 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:57.984442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:57.982811 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.006740 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kj4\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.006890 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.006928 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.006954 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.007010 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.007069 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.007135 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.007231 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.008440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.007266 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.084038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.010081 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:29:58.084038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.030590 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.084038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.075565 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.116672 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.116761 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.116842 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.116874 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.116943 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kj4\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.117069 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.117104 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.117128 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.117848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.117189 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.151326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.124602 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.152550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.151858 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.152550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.152179 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.184846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.158848 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.184846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.159476 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.184846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.159543 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.184846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.168239 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.184846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.170050 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.275194 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.275124 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kj4\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.290135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.290052 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.397455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.381627 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:29:58.560658 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.559555 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.581557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.581506 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:29:58.746719 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.745541 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.894606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.894561 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.895545 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.895520 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:29:58.940445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.936878 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.940445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.937584 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:29:58.940445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.939097 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:58.942165 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:58.942087 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.014538 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.001516 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.014965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.014820 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.015084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.014949 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.041072 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.040964 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.079813 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.086496 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.086600 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.086642 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.086679 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.093674 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.117455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.094064 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:29:59.191440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.190524 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.191440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.191289 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:29:59.561249 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.560061 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:59.562018 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.561991 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:29:59.672440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.635509 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:29:59.672440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.636059 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:29:59.762569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.762517 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:29:59.766867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.766836 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:29:59.913864 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:29:59.913763 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.070549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.070504 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.071529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.071497 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 19:30:00.146451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.142624 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:00.155138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.153920 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:00.345212 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.158997 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:30:00.349441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.159030 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 30 19:30:00.349741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.160556 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 19:30:00.353438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.160898 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.355444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.199101 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.571442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.569072 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:30:00.689630 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.665659 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.796089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.794132 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:00.994500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:00.963112 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.107244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.107098 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.115523 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.115482 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 19:30:01.145458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.143461 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.150454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.146472 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 19:30:01.220439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.182648 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.311714 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.311827 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.311890 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:01.318599 6511 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.173s" Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.318694 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.318738 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.325289 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:30:01.360729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.342989 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl"] Oct 30 19:30:01.383636 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.365708 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:01.383636 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.380281 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:30:01.406453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.394887 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:01.414454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.411256 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl"] Oct 30 19:30:01.414454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.411339 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6"] Oct 30 19:30:01.414454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.402863 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:01.414454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.402826 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:01.499846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.482505 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.596446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.559364 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92pc\" (UniqueName: \"kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc\") pod \"maas-api-key-cleanup-29889810-4t5tl\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:01.597624 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.597579 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-4t5tl\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:01.756327 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.756125 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.756659 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.756635 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92pc\" (UniqueName: \"kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc\") pod \"maas-api-key-cleanup-29889810-4t5tl\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:01.756985 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.756963 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-4t5tl\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:01.802439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.765738 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-4t5tl\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:01.827486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.815121 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" event={"ID":"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652","Type":"ContainerStarted","Data":"36779852c9b048a59c67b00646f01ec1e8c36bf627f2569c23e5e606626986ba"} Oct 30 19:30:01.883899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.883831 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:30:01.909596 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.886783 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 19:30:02.019510 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:01.991479 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.089453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.089367 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92pc\" (UniqueName: \"kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc\") pod \"maas-api-key-cleanup-29889810-4t5tl\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:02.120441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.117901 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.146933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.145205 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.244740 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.244673 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.258546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.258514 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:02.332457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.302817 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:30:02.404733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.404673 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.406021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.405119 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.486960 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.459624 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:02.486960 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.460001 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:02.489587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.489561 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:30:02.604139 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.604090 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.786994 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.763476 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.941470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:02.935506 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.025460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.025395 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.167692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.136149 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.182550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.173223 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.342899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.342842 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.367354 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.367300 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:03.367673 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.367626 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.437475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.428944 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.519504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.492192 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 19:30:03.522780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.522745 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:30:03.582442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.578976 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.582442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.579406 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.623479 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.606447 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.646135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.646086 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:30:03.654163 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.654133 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:30:03.724049 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.723998 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.775026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.725487 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:03.848743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.848697 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:03.875881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:03.875823 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:30:04.025941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.025836 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:04.046622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.046577 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:04.184984 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.182935 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:04.199283 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.199256 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:30:04.200617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.200596 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl"] Oct 30 19:30:04.200830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.200816 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:30:04.201010 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.201000 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:30:04.201185 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.201175 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:30:04.201349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.201340 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:30:04.202160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.202139 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:30:04.202289 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.202246 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:04.204112 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.204092 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:30:04.204749 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.204724 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:30:04.206295 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.206271 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:30:04.219691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.219668 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:30:04.678273 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.678244 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6"] Oct 30 19:30:04.682748 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.682724 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.690883 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.690846 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:04.692714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.692209 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6"] Oct 30 19:30:04.783732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.783699 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.783959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.783809 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.783959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.783903 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gzm\" (UniqueName: \"kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.884745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.884703 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.884928 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.884829 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.884990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.884947 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gzm\" (UniqueName: \"kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.885616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.885593 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.887480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.887460 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:04.894209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:04.894185 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gzm\" (UniqueName: \"kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm\") pod \"maas-api-revoke-sub-3532f2d7-tkjk6\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:05.003959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.003880 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:05.213290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.213249 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" event={"ID":"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652","Type":"ContainerStarted","Data":"3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b"} Oct 30 19:30:05.214779 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.214751 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerStarted","Data":"0eb7718f22e46ad5f8b14620bfb9a3816694ee2706c15fae9a2dbcac225313f4"} Oct 30 19:30:05.248001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.240891 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podStartSLOduration=8.240867567 podStartE2EDuration="8.240867567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.24003176 +0000 UTC m=+779.263635660" watchObservedRunningTime="2026-10-30 19:30:05.240867567 +0000 UTC m=+779.264471470" Oct 30 19:30:05.357448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.356932 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6"] Oct 30 19:30:05.373966 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:30:05.372772 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472d9709_0f07_45c6_af31_34b851c166a5.slice/crio-353470c893997fbf2cb2a01e5360754322f80153834de2bcbd1179afcd2658f4 WatchSource:0}: Error finding container 353470c893997fbf2cb2a01e5360754322f80153834de2bcbd1179afcd2658f4: Status 404 returned error can't find the container with id 353470c893997fbf2cb2a01e5360754322f80153834de2bcbd1179afcd2658f4 Oct 30 19:30:05.383363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.382336 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:30:05.384581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.384555 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:30:05.384756 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:05.384732 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:30:06.247510 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:06.247476 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" event={"ID":"472d9709-0f07-45c6-af31-34b851c166a5","Type":"ContainerStarted","Data":"353470c893997fbf2cb2a01e5360754322f80153834de2bcbd1179afcd2658f4"} Oct 30 19:30:06.386798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:06.384601 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:30:06.386798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:06.384660 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:30:07.382561 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:07.382521 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:30:07.383046 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:07.382574 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:30:08.382980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:08.382656 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:30:08.382980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:08.382704 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:30:09.279039 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:09.278169 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerStarted","Data":"0fdfa9ada0c8e3de65f6afd59874614a43ce4a443a20a0809aeba49680920529"} Oct 30 19:30:09.298054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:09.298003 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" podStartSLOduration=5.110004371 podStartE2EDuration="9.29799058s" totalImagesPullingTime="4.187986209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:09.297637941 +0000 UTC m=+783.321241841" watchObservedRunningTime="2026-10-30 19:30:09.29799058 +0000 UTC m=+783.321594500" Oct 30 19:30:09.382931 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:09.382890 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:30:09.383085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:09.382952 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:30:10.303161 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.303124 6511 generic.go:356] "Generic (PLEG): container finished" podID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerID="0fdfa9ada0c8e3de65f6afd59874614a43ce4a443a20a0809aeba49680920529" exitCode=60 Oct 30 19:30:10.303360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.303212 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerDied","Data":"0fdfa9ada0c8e3de65f6afd59874614a43ce4a443a20a0809aeba49680920529"} Oct 30 19:30:10.303692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.303653 6511 scope.go:122] "RemoveContainer" containerID="0fdfa9ada0c8e3de65f6afd59874614a43ce4a443a20a0809aeba49680920529" Oct 30 19:30:10.305435 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.305386 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" event={"ID":"472d9709-0f07-45c6-af31-34b851c166a5","Type":"ContainerStarted","Data":"41e17ab8c533059747d1bc5dbcbb1c6953716ff3f4bf023fee190b873c5e59a7"} Oct 30 19:30:10.340500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.340343 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" podStartSLOduration=2.336048738 podStartE2EDuration="6.340316002s" totalImagesPullingTime="4.004267264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:10.337917898 +0000 UTC m=+784.361521797" watchObservedRunningTime="2026-10-30 19:30:10.340316002 +0000 UTC m=+784.363919891" Oct 30 19:30:10.382523 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.382487 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:30:10.382680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:10.382545 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:30:11.314423 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.314384 6511 generic.go:356] "Generic (PLEG): container finished" podID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerID="95c7b591998c59863064133a0727ef192ea81d32a8cbb9e49d5e4fc1ef2f314c" exitCode=60 Oct 30 19:30:11.314933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.314503 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerDied","Data":"95c7b591998c59863064133a0727ef192ea81d32a8cbb9e49d5e4fc1ef2f314c"} Oct 30 19:30:11.314933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.314592 6511 scope.go:122] "RemoveContainer" containerID="0fdfa9ada0c8e3de65f6afd59874614a43ce4a443a20a0809aeba49680920529" Oct 30 19:30:11.315125 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.315018 6511 scope.go:122] "RemoveContainer" containerID="95c7b591998c59863064133a0727ef192ea81d32a8cbb9e49d5e4fc1ef2f314c" Oct 30 19:30:11.315470 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:11.315406 6511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-4t5tl_odh-ai-gateway-infra(5bbbb172-48d7-4001-b176-fa64f2e99aea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" Oct 30 19:30:11.317765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.317740 6511 generic.go:356] "Generic (PLEG): container finished" podID="472d9709-0f07-45c6-af31-34b851c166a5" containerID="41e17ab8c533059747d1bc5dbcbb1c6953716ff3f4bf023fee190b873c5e59a7" exitCode=0 Oct 30 19:30:11.317909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.317772 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" event={"ID":"472d9709-0f07-45c6-af31-34b851c166a5","Type":"ContainerDied","Data":"41e17ab8c533059747d1bc5dbcbb1c6953716ff3f4bf023fee190b873c5e59a7"} Oct 30 19:30:11.387359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.387261 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:30:11.387694 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.387655 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:30:11.388380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:11.388359 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:30:12.326814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.326780 6511 scope.go:122] "RemoveContainer" containerID="95c7b591998c59863064133a0727ef192ea81d32a8cbb9e49d5e4fc1ef2f314c" Oct 30 19:30:12.327222 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:12.327118 6511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-4t5tl_odh-ai-gateway-infra(5bbbb172-48d7-4001-b176-fa64f2e99aea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" Oct 30 19:30:12.497376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.497335 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:12.653531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.653446 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm\" (UniqueName: \"kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm\") pod \"472d9709-0f07-45c6-af31-34b851c166a5\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " Oct 30 19:30:12.653531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.653499 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca\") pod \"472d9709-0f07-45c6-af31-34b851c166a5\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " Oct 30 19:30:12.653774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.653604 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca\") pod \"472d9709-0f07-45c6-af31-34b851c166a5\" (UID: \"472d9709-0f07-45c6-af31-34b851c166a5\") " Oct 30 19:30:12.653939 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.653916 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca" pod "472d9709-0f07-45c6-af31-34b851c166a5" (UID: "472d9709-0f07-45c6-af31-34b851c166a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:12.654182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.654162 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-openshift-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:12.655942 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.655918 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm" pod "472d9709-0f07-45c6-af31-34b851c166a5" (UID: "472d9709-0f07-45c6-af31-34b851c166a5"). InnerVolumeSpecName "kube-api-access-72gzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:12.656126 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.656104 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca" pod "472d9709-0f07-45c6-af31-34b851c166a5" (UID: "472d9709-0f07-45c6-af31-34b851c166a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:12.755903 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.755869 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472d9709-0f07-45c6-af31-34b851c166a5-maas-api-xks-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:12.755903 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:12.755904 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72gzm\" (UniqueName: \"kubernetes.io/projected/472d9709-0f07-45c6-af31-34b851c166a5-kube-api-access-72gzm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:13.334981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:13.334952 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" Oct 30 19:30:13.334981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:13.334961 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6" event={"ID":"472d9709-0f07-45c6-af31-34b851c166a5","Type":"ContainerDied","Data":"353470c893997fbf2cb2a01e5360754322f80153834de2bcbd1179afcd2658f4"} Oct 30 19:30:13.335457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:13.335016 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353470c893997fbf2cb2a01e5360754322f80153834de2bcbd1179afcd2658f4" Oct 30 19:30:17.637200 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.637166 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz"] Oct 30 19:30:17.637757 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.637741 6511 cpu_manager.go:414] "containerMap: removing container" podUID="472d9709-0f07-45c6-af31-34b851c166a5" containerName="revoke-keys" Oct 30 19:30:17.637803 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.637759 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472d9709-0f07-45c6-af31-34b851c166a5" containerName="revoke-keys" Oct 30 19:30:17.637877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.637867 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="472d9709-0f07-45c6-af31-34b851c166a5" containerName="revoke-keys" Oct 30 19:30:17.642437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.642393 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.647577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.647555 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:17.650212 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.650189 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz"] Oct 30 19:30:17.722722 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.722684 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc58d\" (UniqueName: \"kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.722892 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.722825 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.722934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.722890 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.824343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.824304 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc58d\" (UniqueName: \"kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.824543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.824392 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.824543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.824494 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.825100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.825077 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.826993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.826971 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.833040 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.833020 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc58d\" (UniqueName: \"kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d\") pod \"maas-api-revoke-sub-0e2e8351-gxgvz\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:17.957471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:17.957367 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:18.237576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:18.237550 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz"] Oct 30 19:30:18.237968 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:30:18.237945 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ddf2a9_d20e_4f0b_9e83_34edf41631b9.slice/crio-170c31d79f61ef96ae0bc3d41c355206c783f54f6ef8965f72b3e90a690a58f5 WatchSource:0}: Error finding container 170c31d79f61ef96ae0bc3d41c355206c783f54f6ef8965f72b3e90a690a58f5: Status 404 returned error can't find the container with id 170c31d79f61ef96ae0bc3d41c355206c783f54f6ef8965f72b3e90a690a58f5 Oct 30 19:30:18.383293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:18.383251 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" event={"ID":"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9","Type":"ContainerStarted","Data":"170c31d79f61ef96ae0bc3d41c355206c783f54f6ef8965f72b3e90a690a58f5"} Oct 30 19:30:19.392141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:19.392106 6511 generic.go:356] "Generic (PLEG): container finished" podID="f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" containerID="16503a573526c15bb1b8b9bc017eb21d68d8f004591c919a50e33c792fe9abf1" exitCode=0 Oct 30 19:30:19.392616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:19.392157 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" event={"ID":"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9","Type":"ContainerDied","Data":"16503a573526c15bb1b8b9bc017eb21d68d8f004591c919a50e33c792fe9abf1"} Oct 30 19:30:20.579445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.577513 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:20.672256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.672170 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca\") pod \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " Oct 30 19:30:20.672256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.672230 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca\") pod \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " Oct 30 19:30:20.672512 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.672352 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d\" (UniqueName: \"kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d\") pod \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\" (UID: \"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9\") " Oct 30 19:30:20.672623 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.672599 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca" pod "f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" (UID: "f2ddf2a9-d20e-4f0b-9e83-34edf41631b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:20.673045 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.673022 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-openshift-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.674616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.674589 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca" pod "f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" (UID: "f2ddf2a9-d20e-4f0b-9e83-34edf41631b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:20.674701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.674648 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d" pod "f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" (UID: "f2ddf2a9-d20e-4f0b-9e83-34edf41631b9"). InnerVolumeSpecName "kube-api-access-tc58d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:20.774191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.774159 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-maas-api-xks-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.774191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:20.774191 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc58d\" (UniqueName: \"kubernetes.io/projected/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9-kube-api-access-tc58d\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.416781 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:21.416744 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" event={"ID":"f2ddf2a9-d20e-4f0b-9e83-34edf41631b9","Type":"ContainerDied","Data":"170c31d79f61ef96ae0bc3d41c355206c783f54f6ef8965f72b3e90a690a58f5"} Oct 30 19:30:21.417011 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:21.416997 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170c31d79f61ef96ae0bc3d41c355206c783f54f6ef8965f72b3e90a690a58f5" Oct 30 19:30:21.417176 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:21.417165 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz" Oct 30 19:30:24.380916 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:24.380878 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:30:26.094355 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:26.094324 6511 scope.go:122] "RemoveContainer" containerID="95c7b591998c59863064133a0727ef192ea81d32a8cbb9e49d5e4fc1ef2f314c" Oct 30 19:30:26.451539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:26.451510 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerStarted","Data":"5fe89f11ee6574c234a9fdbc304092f7bcdcf1dc007eb886c662cfb5954425cb"} Oct 30 19:30:27.255934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.255903 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj"] Oct 30 19:30:27.256715 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.256690 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" containerName="revoke-keys" Oct 30 19:30:27.256715 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.256714 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" containerName="revoke-keys" Oct 30 19:30:27.256908 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.256821 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" containerName="revoke-keys" Oct 30 19:30:27.260749 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.260720 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.278109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.278042 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj"] Oct 30 19:30:27.356970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.356939 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.356970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.356976 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357007 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357056 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxfp\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357189 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357370 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357461 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357498 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.357677 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.357661 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459000 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.458966 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459053 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459070 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459094 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459109 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxfp\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459138 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459247 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459651 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459293 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459651 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459328 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.459651 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.459452 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.460584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.460534 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.460584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.460565 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.460803 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.460780 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.460997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.460971 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.462607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.462582 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.462846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.462817 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.463826 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.463798 6511 generic.go:356] "Generic (PLEG): container finished" podID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerID="5fe89f11ee6574c234a9fdbc304092f7bcdcf1dc007eb886c662cfb5954425cb" exitCode=60 Oct 30 19:30:27.463909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.463828 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerDied","Data":"5fe89f11ee6574c234a9fdbc304092f7bcdcf1dc007eb886c662cfb5954425cb"} Oct 30 19:30:27.463909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.463898 6511 scope.go:122] "RemoveContainer" containerID="95c7b591998c59863064133a0727ef192ea81d32a8cbb9e49d5e4fc1ef2f314c" Oct 30 19:30:27.464267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.464246 6511 scope.go:122] "RemoveContainer" containerID="5fe89f11ee6574c234a9fdbc304092f7bcdcf1dc007eb886c662cfb5954425cb" Oct 30 19:30:27.464602 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:27.464579 6511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-4t5tl_odh-ai-gateway-infra(5bbbb172-48d7-4001-b176-fa64f2e99aea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" Oct 30 19:30:27.468925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.468877 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.469311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.469275 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxfp\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp\") pod \"e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.481944 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.481912 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl"] Oct 30 19:30:27.551243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.551211 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg"] Oct 30 19:30:27.557777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.557751 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.582433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.572570 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg"] Oct 30 19:30:27.602452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.599734 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.678862 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.678971 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679072 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679171 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679203 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679239 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679265 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679312 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.679437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.679349 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25r9\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784397 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784537 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784637 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784668 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784709 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784735 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784785 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784833 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25r9\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.784948 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.785382 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.785829 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.786360 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.786885 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.791036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.788612 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.817457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.808515 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.854434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.847308 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.854434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.849132 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.854434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.850679 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25r9\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9\") pod \"e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:27.882172 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:27.881907 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:28.136875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.080739 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj"] Oct 30 19:30:28.272054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.250582 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:28.272054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.250687 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:28.272054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.250736 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:28.518807 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.511543 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg"] Oct 30 19:30:28.547594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.528571 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:28.547594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.528668 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:28.547594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.528717 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:28.623310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.623219 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" event={"ID":"24facc8d-8e74-4310-9bf7-f7d299467941","Type":"ContainerStarted","Data":"f51a5de753b4f8204dfd7d1e0ec57cdaa3e3418c5c96391e89ab3cca1e7d3905"} Oct 30 19:30:28.637675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:28.637603 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" event={"ID":"f510c1f8-7a68-4839-81d7-f5e952135d2d","Type":"ContainerStarted","Data":"7fddf5fb46de5012372e556e0e5dbac81156f1cf362d26548a187034bd91c8b4"} Oct 30 19:30:29.139856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:29.139797 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:29.393440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:29.383531 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:29.460562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:29.460512 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:29.465796 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:29.465757 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 19:30:29.503626 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:29.503577 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.021646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.021355 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:30:30.240928 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.022065 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:30:30.259447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.027630 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.481230 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.481187 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.493905 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.484050 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 19:30:30.494334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.490501 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.495142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.493488 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.500546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.498218 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:30.500546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.498521 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:30.500546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.499177 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.515563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.515535 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq"] Oct 30 19:30:30.521354 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.519479 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:30.550465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.547333 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:30.613707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.613670 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq"] Oct 30 19:30:30.629724 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.629686 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" event={"ID":"f510c1f8-7a68-4839-81d7-f5e952135d2d","Type":"ContainerStarted","Data":"c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508"} Oct 30 19:30:30.630013 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.629989 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" event={"ID":"24facc8d-8e74-4310-9bf7-f7d299467941","Type":"ContainerStarted","Data":"8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a"} Oct 30 19:30:30.630234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.630216 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:30:30.630434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.630402 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:30.631470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.631452 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:30:30.632277 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.632261 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:30.632479 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.632464 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:30:30.632689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.632674 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:30.632806 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.616522 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:30.632928 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.632908 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:30.633450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.615853 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.658406 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.658336 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-77930c58-openshift-default-dockercfg-zdwbl\"" Oct 30 19:30:30.682709 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.682675 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca\") pod \"5bbbb172-48d7-4001-b176-fa64f2e99aea\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " Oct 30 19:30:30.726537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.726501 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc\" (UniqueName: \"kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc\") pod \"5bbbb172-48d7-4001-b176-fa64f2e99aea\" (UID: \"5bbbb172-48d7-4001-b176-fa64f2e99aea\") " Oct 30 19:30:30.727141 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.727109 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.736439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.728544 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.736638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.736613 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.736795 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.736777 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.736929 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.736911 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.737043 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.737028 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.737253 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.737236 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xzb\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.742371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.737607 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.829632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.829592 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.865481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.683236 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca" pod "5bbbb172-48d7-4001-b176-fa64f2e99aea" (UID: "5bbbb172-48d7-4001-b176-fa64f2e99aea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:30.967784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.932577 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:30.968067 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.968030 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:30.983551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.896159 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:30.983790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.942747 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xzb\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:30.984092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.984070 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.029023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.744606 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podStartSLOduration=3.744588352 podStartE2EDuration="3.744588352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:30.570474718 +0000 UTC m=+804.594078626" watchObservedRunningTime="2026-10-30 19:30:30.744588352 +0000 UTC m=+804.768192253" Oct 30 19:30:31.032508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.032444 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podStartSLOduration=4.032404324 podStartE2EDuration="4.032404324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:30.613749099 +0000 UTC m=+804.637352997" watchObservedRunningTime="2026-10-30 19:30:31.032404324 +0000 UTC m=+805.056008225" Oct 30 19:30:31.054203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.054168 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.054597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.054571 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.055624 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.055585 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.055785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.055767 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.055934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.055915 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.056059 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.056044 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.056184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.056168 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.092482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.062017 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbbb172-48d7-4001-b176-fa64f2e99aea-maas-api-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:31.144560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.059528 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.144767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.059783 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.144855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:30.787453 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.145187 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.058886 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.145324 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.082251 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.145408 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.085511 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.145799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.096200 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.145921 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.129964 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.146302 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.146282 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:30:31.220501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.219279 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xzb\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.254490 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.221963 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" podUID="cd512f66-1e42-4a8d-bd20-0a461fefe100" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.24:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.254490 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.223362 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc" pod "5bbbb172-48d7-4001-b176-fa64f2e99aea" (UID: "5bbbb172-48d7-4001-b176-fa64f2e99aea"). InnerVolumeSpecName "kube-api-access-p92pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:31.281478 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.256688 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p92pc\" (UniqueName: \"kubernetes.io/projected/5bbbb172-48d7-4001-b176-fa64f2e99aea-kube-api-access-p92pc\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:31.304707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.304669 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.305327 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.305272 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.481761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.479686 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.481761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.480097 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:30:31.513860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.511864 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.513860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.511993 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:30:31.513860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.512039 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 19:30:31.513860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.512092 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.513860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.512135 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:31.515385 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.514643 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:30:31.529855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.519046 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:30:31.529855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.519139 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:30:31.529855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.519176 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:30:31.529855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.522809 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:30:31.572038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.571856 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" Oct 30 19:30:31.576115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.575454 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl" event={"ID":"5bbbb172-48d7-4001-b176-fa64f2e99aea","Type":"ContainerDied","Data":"0eb7718f22e46ad5f8b14620bfb9a3816694ee2706c15fae9a2dbcac225313f4"} Oct 30 19:30:31.576115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.575514 6511 scope.go:122] "RemoveContainer" containerID="5fe89f11ee6574c234a9fdbc304092f7bcdcf1dc007eb886c662cfb5954425cb" Oct 30 19:30:31.619339 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.619165 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:31.619339 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.619313 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:31.657149 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.657113 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token\") pod \"e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.678628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.678598 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:31.780052 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.779986 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:31.889652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.889592 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:31.889953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:31.889899 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:32.148727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.148664 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.161476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.158534 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:30:32.336911 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.336253 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:30:32.508442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.473543 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.508442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.489682 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.508442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.490362 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:30:32.508442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.507540 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.581462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.518509 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:32.581462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.533493 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.581462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.533858 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.581462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.562950 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:30:32.581462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.578197 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.581462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.581041 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:32.621922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.621831 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:32.622166 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.622128 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:32.711444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.698880 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:32.928498 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.903580 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:32.928879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:32.928835 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:33.178437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.178140 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:30:33.419442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.418570 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 19:30:33.520443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.494090 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:33.559443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.501655 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 19:30:33.881759 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.648531 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:33.906480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.668434 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:33.927505 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.720350 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.730729 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.730783 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.730885 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.740527 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.741534 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.741634 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.741908 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:33.886645 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:30:34.142528 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.078767 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:34.262621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.179058 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.262621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.236432 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.262621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.237216 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:34.262621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.237293 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:34.431649 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.431543 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.645440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.644826 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:30:34.660445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.656563 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.760491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.690547 6511 patch_prober.go:29] interesting pod/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:34.760491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.690622 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:34.903880 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.890993 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:34.928926 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.928882 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:34.953443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.941085 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:30:34.953443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.941497 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:30:34.977952 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:34.954168 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 19:30:35.010223 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.009857 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:35.010223 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.009933 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:35.021000 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.020878 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:35.021292 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.021264 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:35.021488 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.021466 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:30:35.171238 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.160166 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:30:35.266203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.266152 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:35.323461 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:35.282082 6511 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.135s" Oct 30 19:30:35.324208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.324177 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl"] Oct 30 19:30:35.338444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.324322 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-4t5tl"] Oct 30 19:30:35.338784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.338757 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:35.339153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.339124 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 19:30:35.339560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.339534 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:30:35.353437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.343547 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 19:30:35.373441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.353918 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:30:35.486034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.386605 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:30:35.607527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.567667 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:30:35.616252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.616214 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 19:30:35.706078 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.706044 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq"] Oct 30 19:30:35.712612 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.712583 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:35.712848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.712822 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:35.791055 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.790694 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:35.792848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.792825 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:30:35.826442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.811100 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:30:35.826442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.814064 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:30:35.905668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.899493 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:30:35.922447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.916023 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:30:35.922447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.917159 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:30:35.922447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.917971 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:30:35.996636 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.996577 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:35.998538 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.998500 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:36.001837 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:35.999496 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:30:36.001837 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.001594 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:30:36.003957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.003725 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:30:36.004123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.004097 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.004209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.004144 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:36.010439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.006515 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:30:36.088653 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.088617 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:30:36.178534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.172486 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" path="/var/lib/kubelet/pods/5bbbb172-48d7-4001-b176-fa64f2e99aea/volumes" Oct 30 19:30:36.398472 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.397803 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" event={"ID":"a446f6c6-8061-432d-97df-a2065ec683af","Type":"ContainerStarted","Data":"037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b"} Oct 30 19:30:36.401550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.401522 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" event={"ID":"a446f6c6-8061-432d-97df-a2065ec683af","Type":"ContainerStarted","Data":"0aee3aaf53251f0e37fc98b807024276ff2a3b584831b9aa5f8d8e56dba14c56"} Oct 30 19:30:36.655476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.610910 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.695457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.675476 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:36.728303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.728244 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:36.896441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.895775 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.896441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.895843 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:36.928785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:36.928665 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podStartSLOduration=7.928643497 podStartE2EDuration="7.928643497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:36.849045334 +0000 UTC m=+810.872649252" watchObservedRunningTime="2026-10-30 19:30:36.928643497 +0000 UTC m=+810.952247397" Oct 30 19:30:37.609503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:37.608658 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.609503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:37.608711 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:37.711340 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:37.711307 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:37.887653 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:37.883631 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.887653 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:37.883706 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:38.427446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:38.424528 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:38.432454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:38.429656 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:30:38.615174 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:38.615113 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:38.615498 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:38.615464 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:38.893440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:38.884732 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:38.893440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:38.884792 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:39.607905 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:39.607876 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:39.608137 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:39.608109 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:39.883869 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:39.883758 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:30:39.883869 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:39.883826 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:30:40.610185 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:40.609226 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 19:30:40.610185 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:40.609312 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 19:30:41.552312 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:41.552269 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:41.553798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:41.553774 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:41.556098 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:41.556074 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:30:41.911345 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:41.911248 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:41.911820 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:41.911639 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:41.912482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:41.912457 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:30:50.290139 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.287104 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk"] Oct 30 19:30:50.290616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290333 6511 cpu_manager.go:414] "containerMap: removing container" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290358 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290388 6511 cpu_manager.go:414] "containerMap: removing container" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290397 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290457 6511 cpu_manager.go:414] "containerMap: removing container" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290466 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290652 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.290923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.290681 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:30:50.304497 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.298200 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.318042 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.318009 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-77930c58-serving-cert\"" Oct 30 19:30:50.318530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.318507 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk"] Oct 30 19:30:50.429092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.428534 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.429092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.428686 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.429092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.428734 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2vd\" (UniqueName: \"kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.429092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.428791 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.537151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.537115 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.537389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.537178 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2vd\" (UniqueName: \"kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.537389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.537225 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.537685 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.537459 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.538403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.538339 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.540549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.540491 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.540652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.540578 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.560896 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.560856 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2vd\" (UniqueName: \"kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd\") pod \"maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:50.640930 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:50.640883 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:51.081755 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:51.073264 6511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:30:51.081755 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:51.073634 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk"] Oct 30 19:30:51.638724 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:51.638656 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" event={"ID":"733f7b67-06ae-46b5-b89c-4f7fc634f789","Type":"ContainerStarted","Data":"53ba405686b0190dfe65ab941d30b3a4ccd455a1acd703c5abcbeb9f98bf880c"} Oct 30 19:30:52.647331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:52.647296 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" event={"ID":"733f7b67-06ae-46b5-b89c-4f7fc634f789","Type":"ContainerStarted","Data":"5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c"} Oct 30 19:30:52.647866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:52.647463 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:30:52.668150 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:52.668088 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" podStartSLOduration=2.104651017 podStartE2EDuration="2.668073128s" totalImagesPullingTime="563.422111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:52.665211516 +0000 UTC m=+826.688815414" watchObservedRunningTime="2026-10-30 19:30:52.668073128 +0000 UTC m=+826.691677027" Oct 30 19:30:54.382287 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:30:54.382254 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:30:58.665071 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:30:58.665040 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:31:05.505444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.492545 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.534192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.529743 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:31:05.570543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.570495 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:05.619593 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.619537 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.707603 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.707553 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.772626 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.772515 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 19:31:05.873508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.873458 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:05.926549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:05.926502 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 19:31:06.127441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.126489 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 19:31:06.140650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.135010 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 19:31:06.140650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.137107 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.140650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.137182 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:06.216078 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.214152 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.281581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.281527 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" podUID="cd512f66-1e42-4a8d-bd20-0a461fefe100" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.24:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.501878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.501579 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:06.530575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.530534 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.555977 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.555925 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:06.580044 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.579943 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.619252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.619220 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:31:06.640680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.640645 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:31:06.640875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.640710 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:31:06.641457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.619965 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:31:06.641986 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.641927 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:31:06.715167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.715125 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:06.715846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.715814 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:31:06.931446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.928519 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:31:06.931446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.928841 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 19:31:06.931446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.929587 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:31:06.975263 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.963909 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 19:31:07.004633 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:06.975910 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:31:07.170457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.138674 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.170457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.139120 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:31:07.194404 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.194314 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.195043 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.195020 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:31:07.644663 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.644608 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:31:07.673454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.646982 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.673454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.647293 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:31:07.673454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.663802 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:07.673454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.664572 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.833453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.804857 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.931583 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.927751 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:07.950454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:07.933569 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 19:31:08.082947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.082904 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.144466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.139666 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.155499 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.151740 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.221967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.221861 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:08.249574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.244487 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.650457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.645509 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.655702 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.655662 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.677020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.676969 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.679770 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.678190 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.809084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.807774 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.929908 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.929808 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:08.932112 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.932082 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:31:08.933793 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:08.933767 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.127441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.014716 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:31:09.127441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.043552 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.127441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.119536 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-z6m7s" podUID="cd512f66-1e42-4a8d-bd20-0a461fefe100" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.24:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.146684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.144036 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.146684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.145483 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:31:09.225193 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.225092 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.252597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.226540 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.255614 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.255554 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.410216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.398803 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:31:09.652568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.652517 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.718439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.693900 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.923459 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.806220 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.924357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.817704 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.924791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.836711 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:09.958454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.958392 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:09.959732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:09.959697 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:10.086439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.050598 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Oct 30 19:31:10.089876 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.089830 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.090628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.090583 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 30 19:31:10.153387 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.153341 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.259753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.259644 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.303442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.291653 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:31:10.303442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.295273 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.702440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.701462 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.845571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.845519 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.925576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.925516 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.926973 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.926937 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.928432 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.928085 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.928855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.928112 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:31:10.966942 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.966846 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:10.985750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:10.985710 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.094479 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.094387 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 19:31:11.097455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.095636 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:31:11.165441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.164547 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 19:31:11.296946 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.296842 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.298330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.298297 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.298860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.298531 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.703112 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.703065 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.796497 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.796449 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.926574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.926526 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.928266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.928236 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:31:11.928736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.928697 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.929239 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.929095 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.969641 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.969536 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:11.996568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:11.996516 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.098544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.098495 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 19:31:12.174668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.174625 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.302643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.302596 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.303034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.302878 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.303559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.302910 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:12.483542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.483505 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:12.483893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.483799 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.734599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.707609 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.928593 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.928528 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:12.929487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.929456 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.929741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.929716 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:12.931549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.931526 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:31:12.980440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:12.971674 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.026498 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.009721 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.117012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.116967 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.177581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.177531 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.243564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.243515 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.334442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.329597 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:31:13.334442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.330002 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 19:31:13.334442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.331723 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.604494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.599885 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded" Oct 30 19:31:13.714157 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.713031 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 19:31:13.941518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.939494 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.941518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.939884 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:13.941518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.940136 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 19:31:13.963101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.960679 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:31:13.977695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.977656 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:13.987848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:13.987491 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:14.007455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.007410 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:31:14.034458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.023491 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:14.075484 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.075454 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:31:14.075777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.075762 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:31:14.178587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.178548 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:14.293698 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.293615 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:31:14.293973 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.293954 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:31:14.294152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.294139 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:31:14.294290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.294279 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:31:14.295969 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.295924 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:31:14.323089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.323058 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:31:14.411448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.407721 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:31:14.462275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:14.459817 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:31:20.024005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.023969 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2"] Oct 30 19:31:20.025350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.025323 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbbb172-48d7-4001-b176-fa64f2e99aea" containerName="cleanup" Oct 30 19:31:20.030874 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.030852 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.045064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.044963 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5e333f16-serving-cert\"" Oct 30 19:31:20.059539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.059511 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2"] Oct 30 19:31:20.132295 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.132266 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86pwj\" (UniqueName: \"kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.132463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.132315 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.132463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.132409 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.132560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.132517 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.234670 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.234631 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.235011 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.234989 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.235192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.235179 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.236314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.235517 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.236704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.236684 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86pwj\" (UniqueName: \"kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.238389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.238361 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.238754 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.238732 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.251078 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.251050 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86pwj\" (UniqueName: \"kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj\") pod \"maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.364650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.364620 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:20.672866 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:20.672835 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2"] Oct 30 19:31:20.676550 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:31:20.676523 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8974ac9_9fb3_47cb_8c36_d38f0b1f8467.slice/crio-ff4474a624f2ef2d29cf906b14a7249350f8e7ec01b1d3257f35cfbfb915b814 WatchSource:0}: Error finding container ff4474a624f2ef2d29cf906b14a7249350f8e7ec01b1d3257f35cfbfb915b814: Status 404 returned error can't find the container with id ff4474a624f2ef2d29cf906b14a7249350f8e7ec01b1d3257f35cfbfb915b814 Oct 30 19:31:21.393086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.393053 6511 generic.go:356] "Generic (PLEG): container finished" podID="a446f6c6-8061-432d-97df-a2065ec683af" containerID="037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b" exitCode=0 Oct 30 19:31:21.393619 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.393128 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" event={"ID":"a446f6c6-8061-432d-97df-a2065ec683af","Type":"ContainerDied","Data":"037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b"} Oct 30 19:31:21.393749 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.393727 6511 scope.go:122] "RemoveContainer" containerID="037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b" Oct 30 19:31:21.395395 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.395369 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" event={"ID":"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467","Type":"ContainerStarted","Data":"9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f"} Oct 30 19:31:21.395545 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.395464 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:21.395545 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.395491 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" event={"ID":"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467","Type":"ContainerStarted","Data":"ff4474a624f2ef2d29cf906b14a7249350f8e7ec01b1d3257f35cfbfb915b814"} Oct 30 19:31:21.395662 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.395536 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:21.395662 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.395623 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:21.395771 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.395678 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:21.438666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:21.438340 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" podStartSLOduration=1.929928101 podStartE2EDuration="2.438322098s" totalImagesPullingTime="508.393997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:21.436782936 +0000 UTC m=+855.460386839" watchObservedRunningTime="2026-10-30 19:31:21.438322098 +0000 UTC m=+855.461925998" Oct 30 19:31:22.447480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:22.447443 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" event={"ID":"a446f6c6-8061-432d-97df-a2065ec683af","Type":"ContainerStarted","Data":"25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b"} Oct 30 19:31:22.704917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:22.704825 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:31:22.706300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:22.706272 6511 patch_prober.go:29] interesting pod/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:31:22.706498 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:22.706342 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 19:31:23.429438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:23.427510 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:31:23.705589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:23.705520 6511 patch_prober.go:29] interesting pod/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:31:23.705589 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:23.705573 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 19:31:24.382645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.382568 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf"] Oct 30 19:31:24.401455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.392170 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.401455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.393822 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf"] Oct 30 19:31:24.401455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.399112 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:31:24.401455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.399167 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:31:24.401455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.399252 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h9bmh\"" Oct 30 19:31:24.401455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.399336 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:31:24.423338 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:31:24.423294 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:31:24.480464 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.457538 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.483906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.483851 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756m6\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.484357 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.484324 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.486599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.484633 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.486599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.484693 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.486599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.484809 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.486599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.484900 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.586530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.586494 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.586705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.586587 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756m6\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.586705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.586695 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.586787 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.586755 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.586831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.586800 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.586948 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.586930 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.587517 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.587299 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.587623 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.587545 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.589965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.589931 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.590231 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.590211 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.590346 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.590325 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.590782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.590760 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.596565 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.596531 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.596987 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.596964 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756m6\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2zxhf\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:24.705092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.704998 6511 patch_prober.go:29] interesting pod/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:31:24.705092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.705057 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 19:31:24.789004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:24.788970 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:25.149447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:25.117667 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf"] Oct 30 19:31:25.502849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:25.502770 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" event={"ID":"4c25a6ef-b4b2-4e36-b3cc-847220439ec7","Type":"ContainerStarted","Data":"f3290946c0968b7fee4dd3d75589c8415d4bc1ce47d43255e4e0c4ec2b9c9587"} Oct 30 19:31:25.715440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:25.708873 6511 patch_prober.go:29] interesting pod/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:31:25.715440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:25.708959 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 19:31:26.705703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:26.705673 6511 patch_prober.go:29] interesting pod/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:31:26.706160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:26.705728 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 19:31:27.459573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.455888 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:31:27.556465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.555850 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9"] Oct 30 19:31:27.562307 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.562278 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9"] Oct 30 19:31:27.562482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.562463 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569196 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569252 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569332 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569432 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtrq\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569567 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569604 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.569807 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569785 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.570022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.569997 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.570129 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.570106 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.671994 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672056 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672082 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672176 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672251 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtrq\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672291 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672324 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.672487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672399 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.673169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.672580 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.673354 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.673330 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.673675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.673654 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.674567 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.674543 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.675089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.675061 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.675328 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.675306 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.677321 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.677293 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.678879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.678851 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.684322 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.684298 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.684867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.684840 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtrq\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq\") pod \"e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:27.705714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.705667 6511 patch_prober.go:29] interesting pod/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 19:31:27.706120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.705721 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 19:31:27.892199 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:27.891820 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:28.237187 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.237113 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9"] Oct 30 19:31:28.242224 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:31:28.242184 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ffe36d_f98a_4600_8b0e_7366f8196de9.slice/crio-285227f000ad8588a62252424700753e31f61115390ab0bd9885db0f2051e3ae WatchSource:0}: Error finding container 285227f000ad8588a62252424700753e31f61115390ab0bd9885db0f2051e3ae: Status 404 returned error can't find the container with id 285227f000ad8588a62252424700753e31f61115390ab0bd9885db0f2051e3ae Oct 30 19:31:28.245938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.245389 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:28.245938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.245504 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:28.245938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.245546 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:28.572323 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.572286 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" event={"ID":"69ffe36d-f98a-4600-8b0e-7366f8196de9","Type":"ContainerStarted","Data":"3332dc068fd49e07e5174902be6d2105ecfb22bde047b70a1b430e7e50419bbf"} Oct 30 19:31:28.572533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.572352 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" event={"ID":"69ffe36d-f98a-4600-8b0e-7366f8196de9","Type":"ContainerStarted","Data":"285227f000ad8588a62252424700753e31f61115390ab0bd9885db0f2051e3ae"} Oct 30 19:31:28.597689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.597630 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podStartSLOduration=1.597611595 podStartE2EDuration="1.597611595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:28.595585164 +0000 UTC m=+862.619189066" watchObservedRunningTime="2026-10-30 19:31:28.597611595 +0000 UTC m=+862.621215496" Oct 30 19:31:28.711070 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.711038 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:31:28.712746 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.712724 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:31:28.897492 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:28.894521 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:29.895132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:29.895092 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:31:29.895562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:29.895161 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 19:31:30.267252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.267214 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:30.267350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.267304 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:31:30.592718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.592681 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" event={"ID":"4c25a6ef-b4b2-4e36-b3cc-847220439ec7","Type":"ContainerStarted","Data":"e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc"} Oct 30 19:31:30.592919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.592884 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:30.616695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.616634 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" podStartSLOduration=1.516057532 podStartE2EDuration="6.616615669s" totalImagesPullingTime="5.100558137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:30.615371232 +0000 UTC m=+864.638975130" watchObservedRunningTime="2026-10-30 19:31:30.616615669 +0000 UTC m=+864.640219569" Oct 30 19:31:30.895782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.895683 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:31:30.896268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:30.895800 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 19:31:31.342441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.334209 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk"] Oct 30 19:31:31.478443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.473897 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" podUID="733f7b67-06ae-46b5-b89c-4f7fc634f789" containerName="maas-api" containerID="cri-o://5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c" gracePeriod=30 Oct 30 19:31:31.725437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.722028 6511 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2zxhf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:31:31.725437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.722120 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" podUID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:31:31.858461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.851950 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq"] Oct 30 19:31:31.858461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.852257 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" containerID="cri-o://25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b" gracePeriod=30 Oct 30 19:31:31.903051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.901549 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:31:31.903051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:31.901629 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 19:31:32.708556 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.708526 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:31:32.710776 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.710749 6511 generic.go:356] "Generic (PLEG): container finished" podID="733f7b67-06ae-46b5-b89c-4f7fc634f789" containerID="5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c" exitCode=0 Oct 30 19:31:32.710882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.710785 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" Oct 30 19:31:32.710882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.710813 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" event={"ID":"733f7b67-06ae-46b5-b89c-4f7fc634f789","Type":"ContainerDied","Data":"5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c"} Oct 30 19:31:32.710993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.710851 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk" event={"ID":"733f7b67-06ae-46b5-b89c-4f7fc634f789","Type":"ContainerDied","Data":"53ba405686b0190dfe65ab941d30b3a4ccd455a1acd703c5abcbeb9f98bf880c"} Oct 30 19:31:32.710993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.710874 6511 scope.go:122] "RemoveContainer" containerID="5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c" Oct 30 19:31:32.758059 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.758033 6511 scope.go:122] "RemoveContainer" containerID="5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c" Oct 30 19:31:32.758507 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:31:32.758482 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c\": container with ID starting with 5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c not found: ID does not exist" containerID="5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c" Oct 30 19:31:32.758616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.758519 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c"} err="failed to get container status \"5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c\": rpc error: code = NotFound desc = could not find container \"5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c\": container with ID starting with 5e5dcc08d2e9356b66892af9897bf555513a463dd2f251af3d509c3c6ad8613c not found: ID does not exist" Oct 30 19:31:32.856204 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.856109 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca\") pod \"733f7b67-06ae-46b5-b89c-4f7fc634f789\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " Oct 30 19:31:32.856204 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.856177 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd\" (UniqueName: \"kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd\") pod \"733f7b67-06ae-46b5-b89c-4f7fc634f789\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " Oct 30 19:31:32.856458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.856289 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs\") pod \"733f7b67-06ae-46b5-b89c-4f7fc634f789\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " Oct 30 19:31:32.856458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.856351 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls\") pod \"733f7b67-06ae-46b5-b89c-4f7fc634f789\" (UID: \"733f7b67-06ae-46b5-b89c-4f7fc634f789\") " Oct 30 19:31:32.856581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.856558 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca" pod "733f7b67-06ae-46b5-b89c-4f7fc634f789" (UID: "733f7b67-06ae-46b5-b89c-4f7fc634f789"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:32.857494 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.857469 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733f7b67-06ae-46b5-b89c-4f7fc634f789-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:32.859246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.859221 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd" pod "733f7b67-06ae-46b5-b89c-4f7fc634f789" (UID: "733f7b67-06ae-46b5-b89c-4f7fc634f789"). InnerVolumeSpecName "kube-api-access-pd2vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:32.859454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.859436 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls" pod "733f7b67-06ae-46b5-b89c-4f7fc634f789" (UID: "733f7b67-06ae-46b5-b89c-4f7fc634f789"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:32.862159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.862135 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs" pod "733f7b67-06ae-46b5-b89c-4f7fc634f789" (UID: "733f7b67-06ae-46b5-b89c-4f7fc634f789"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:32.894177 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.894142 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.894343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.894197 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 19:31:32.959823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.959738 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2vd\" (UniqueName: \"kubernetes.io/projected/733f7b67-06ae-46b5-b89c-4f7fc634f789-kube-api-access-pd2vd\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:32.959823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.959778 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:32.959823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:32.959793 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733f7b67-06ae-46b5-b89c-4f7fc634f789-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:33.114770 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:33.114654 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk"] Oct 30 19:31:33.148456 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:33.145492 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77930c58-7b8cb8d6bc-f97tk"] Oct 30 19:31:33.892757 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:33.892719 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.892913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:33.892775 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 19:31:34.098436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:34.098393 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733f7b67-06ae-46b5-b89c-4f7fc634f789" path="/var/lib/kubelet/pods/733f7b67-06ae-46b5-b89c-4f7fc634f789/volumes" Oct 30 19:31:34.703892 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:34.703858 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:31:34.893294 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:34.893256 6511 patch_prober.go:29] interesting pod/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:31:34.893503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:34.893326 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 19:31:36.304438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:36.304166 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:36.304829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:36.304759 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:36.305526 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:36.305502 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:31:37.291606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.291578 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:31:37.459789 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.459709 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.459789 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.459768 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.459846 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.459878 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.459923 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460033 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:37.460248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460086 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:37.460248 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460170 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460266 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460295 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460324 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data\") pod \"a446f6c6-8061-432d-97df-a2065ec683af\" (UID: \"a446f6c6-8061-432d-97df-a2065ec683af\") " Oct 30 19:31:37.460729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460354 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:37.460941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460819 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:37.460941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.460892 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:37.461543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.461515 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.461675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.461546 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a446f6c6-8061-432d-97df-a2065ec683af-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.461675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.461561 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.461675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.461575 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.461675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.461590 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.462366 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.462341 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:37.462366 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.462349 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:31:37.462572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.462469 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:37.462611 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.462601 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb" pod "a446f6c6-8061-432d-97df-a2065ec683af" (UID: "a446f6c6-8061-432d-97df-a2065ec683af"). InnerVolumeSpecName "kube-api-access-p9xzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:37.562731 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.562698 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9xzb\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-kube-api-access-p9xzb\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.562731 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.562729 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a446f6c6-8061-432d-97df-a2065ec683af-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.562931 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.562740 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a446f6c6-8061-432d-97df-a2065ec683af-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.562931 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.562751 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a446f6c6-8061-432d-97df-a2065ec683af-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:37.782344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.782262 6511 generic.go:356] "Generic (PLEG): container finished" podID="a446f6c6-8061-432d-97df-a2065ec683af" containerID="25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b" exitCode=0 Oct 30 19:31:37.782344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.782312 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" Oct 30 19:31:37.782595 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.782338 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" event={"ID":"a446f6c6-8061-432d-97df-a2065ec683af","Type":"ContainerDied","Data":"25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b"} Oct 30 19:31:37.782595 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.782392 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq" event={"ID":"a446f6c6-8061-432d-97df-a2065ec683af","Type":"ContainerDied","Data":"0aee3aaf53251f0e37fc98b807024276ff2a3b584831b9aa5f8d8e56dba14c56"} Oct 30 19:31:37.782595 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.782429 6511 scope.go:122] "RemoveContainer" containerID="25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b" Oct 30 19:31:37.794044 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.794027 6511 scope.go:122] "RemoveContainer" containerID="037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b" Oct 30 19:31:37.816643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.816605 6511 scope.go:122] "RemoveContainer" containerID="25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b" Oct 30 19:31:37.820736 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:31:37.819813 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b\": container with ID starting with 25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b not found: ID does not exist" containerID="25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b" Oct 30 19:31:37.820736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.819871 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b"} err="failed to get container status \"25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b\": rpc error: code = NotFound desc = could not find container \"25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b\": container with ID starting with 25e2f3ecd594af8e9bfca5b8d1b4a826bcb64173790acb6a27e2e19d1e86711b not found: ID does not exist" Oct 30 19:31:37.820736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.819897 6511 scope.go:122] "RemoveContainer" containerID="037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b" Oct 30 19:31:37.820736 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:31:37.820296 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b\": container with ID starting with 037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b not found: ID does not exist" containerID="037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b" Oct 30 19:31:37.820736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.820331 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b"} err="failed to get container status \"037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b\": rpc error: code = NotFound desc = could not find container \"037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b\": container with ID starting with 037c26e2f9d359f4fa9612a7919ce133838aac356a00f95c1c33c1d711d51b3b not found: ID does not exist" Oct 30 19:31:37.823469 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.823401 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq"] Oct 30 19:31:37.827936 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:37.827913 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-77930c58-openshift-default-5d78dcfbdf-hcqvq"] Oct 30 19:31:38.098224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:38.098185 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a446f6c6-8061-432d-97df-a2065ec683af" path="/var/lib/kubelet/pods/a446f6c6-8061-432d-97df-a2065ec683af/volumes" Oct 30 19:31:48.665559 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:48.665510 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 19:31:48.666621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:48.666588 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 19:31:49.016901 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:48.972128 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:31:49.016901 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:48.987164 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.249684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:49.249640 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.043451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:31:50.033187 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:54.378699 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:31:54.378660 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:32:01.004267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.004225 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2"] Oct 30 19:32:01.004842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.004512 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" podUID="e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" containerName="maas-api" containerID="cri-o://9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f" gracePeriod=30 Oct 30 19:32:01.602706 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.602671 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:32:01.610330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.610296 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs\") pod \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " Oct 30 19:32:01.610520 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.610351 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls\") pod \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " Oct 30 19:32:01.610520 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.610399 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca\") pod \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " Oct 30 19:32:01.610652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.610548 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj\" (UniqueName: \"kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj\") pod \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\" (UID: \"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467\") " Oct 30 19:32:01.610882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.610848 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca" pod "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" (UID: "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:01.611224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.611197 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:01.613083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.613042 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls" pod "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" (UID: "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:01.613210 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.613189 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs" pod "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" (UID: "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:01.613277 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.613241 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj" pod "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" (UID: "e8974ac9-9fb3-47cb-8c36-d38f0b1f8467"). InnerVolumeSpecName "kube-api-access-86pwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:01.699391 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.699164 6511 generic.go:356] "Generic (PLEG): container finished" podID="e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" containerID="9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f" exitCode=0 Oct 30 19:32:01.699391 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.699249 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" Oct 30 19:32:01.699391 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.699253 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" event={"ID":"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467","Type":"ContainerDied","Data":"9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f"} Oct 30 19:32:01.699701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.699408 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2" event={"ID":"e8974ac9-9fb3-47cb-8c36-d38f0b1f8467","Type":"ContainerDied","Data":"ff4474a624f2ef2d29cf906b14a7249350f8e7ec01b1d3257f35cfbfb915b814"} Oct 30 19:32:01.699701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.699457 6511 scope.go:122] "RemoveContainer" containerID="9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f" Oct 30 19:32:01.713858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.713818 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:01.714021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.714006 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:01.714120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.714106 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86pwj\" (UniqueName: \"kubernetes.io/projected/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467-kube-api-access-86pwj\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:01.717848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.717764 6511 scope.go:122] "RemoveContainer" containerID="9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f" Oct 30 19:32:01.718148 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:01.718124 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f\": container with ID starting with 9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f not found: ID does not exist" containerID="9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f" Oct 30 19:32:01.718201 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.718162 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f"} err="failed to get container status \"9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f\": rpc error: code = NotFound desc = could not find container \"9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f\": container with ID starting with 9845cd33937e449475fc8cabfffcadedc3ee93cc1f68f9609b2b4405fc15a04f not found: ID does not exist" Oct 30 19:32:01.743075 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.743045 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2"] Oct 30 19:32:01.748732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:01.748692 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5e333f16-78b65d8cdd-tnns2"] Oct 30 19:32:02.113439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:02.113370 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" path="/var/lib/kubelet/pods/e8974ac9-9fb3-47cb-8c36-d38f0b1f8467/volumes" Oct 30 19:32:06.264752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:06.264722 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:32:06.268040 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:06.267829 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:32:14.367425 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.367379 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s"] Oct 30 19:32:14.368152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368131 6511 cpu_manager.go:414] "containerMap: removing container" podUID="e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" containerName="maas-api" Oct 30 19:32:14.368246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368154 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" containerName="maas-api" Oct 30 19:32:14.368246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368181 6511 cpu_manager.go:414] "containerMap: removing container" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" Oct 30 19:32:14.368246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368189 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" Oct 30 19:32:14.368246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368238 6511 cpu_manager.go:414] "containerMap: removing container" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" Oct 30 19:32:14.368246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368245 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" Oct 30 19:32:14.368542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368257 6511 cpu_manager.go:414] "containerMap: removing container" podUID="733f7b67-06ae-46b5-b89c-4f7fc634f789" containerName="maas-api" Oct 30 19:32:14.368542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368263 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733f7b67-06ae-46b5-b89c-4f7fc634f789" containerName="maas-api" Oct 30 19:32:14.368542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368439 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" Oct 30 19:32:14.368542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368452 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="733f7b67-06ae-46b5-b89c-4f7fc634f789" containerName="maas-api" Oct 30 19:32:14.368542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.368468 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8974ac9-9fb3-47cb-8c36-d38f0b1f8467" containerName="maas-api" Oct 30 19:32:14.373592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.373572 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.380326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.380303 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s"] Oct 30 19:32:14.382632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.381277 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2fd9a937-serving-cert\"" Oct 30 19:32:14.413491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.413459 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.413635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.413620 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vlb\" (UniqueName: \"kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.413682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.413668 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.413729 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.413714 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.515602 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.515567 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vlb\" (UniqueName: \"kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.518115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.518087 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.518276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.518120 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.518276 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:14.518253 6511 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-serving-cert: secret "maas-api-e2e-mt-2fd9a937-serving-cert" not found Oct 30 19:32:14.518405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.518303 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.518405 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:14.518340 6511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls podName:867374a3-3a43-411e-9f98-12ae05c958d4 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:15.01831528 +0000 UTC m=+909.041919180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls") pod "maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" (UID: "867374a3-3a43-411e-9f98-12ae05c958d4") : secret "maas-api-e2e-mt-2fd9a937-serving-cert" not found Oct 30 19:32:14.519302 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.519275 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.522458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.521537 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:14.528993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:14.528969 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vlb\" (UniqueName: \"kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:15.026242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:15.026204 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:15.029108 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:15.029070 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\") pod \"maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:15.291272 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:15.291169 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:15.617088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:15.613385 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s"] Oct 30 19:32:15.825128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:15.825089 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" event={"ID":"867374a3-3a43-411e-9f98-12ae05c958d4","Type":"ContainerStarted","Data":"45049bc73b369dcfc5acde28549805ec8dbb1e4e6e02a2e67750dbf81ce2c501"} Oct 30 19:32:16.833607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:16.833573 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" event={"ID":"867374a3-3a43-411e-9f98-12ae05c958d4","Type":"ContainerStarted","Data":"8c4577b0312e2373b29eebd628d3e5fe27dda56571196763470b59699e4e2d3a"} Oct 30 19:32:16.834007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:16.833643 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:16.854465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:16.854406 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" podStartSLOduration=2.450900491 podStartE2EDuration="2.854393652s" totalImagesPullingTime="403.493161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:16.85246312 +0000 UTC m=+910.876067019" watchObservedRunningTime="2026-10-30 19:32:16.854393652 +0000 UTC m=+910.877997550" Oct 30 19:32:17.799214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.799178 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg"] Oct 30 19:32:17.799945 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.799922 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a446f6c6-8061-432d-97df-a2065ec683af" containerName="istio-proxy" Oct 30 19:32:17.805783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.805752 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.816969 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.816930 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-ae425f-serving-cert\"" Oct 30 19:32:17.822750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.821136 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg"] Oct 30 19:32:17.872339 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.872301 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.872704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.872404 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbtf\" (UniqueName: \"kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.872704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.872550 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.872775 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.872703 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.974657 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.974619 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.974826 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:17.974735 6511 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-serving-cert: secret "maas-api-e2e-shared-b-w0-ae425f-serving-cert" not found Oct 30 19:32:17.974826 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.974775 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.974826 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:17.974803 6511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls podName:31558791-85f3-4319-be66-e9c51095b1e3 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:18.474780209 +0000 UTC m=+912.498384102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls") pod "maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" (UID: "31558791-85f3-4319-be66-e9c51095b1e3") : secret "maas-api-e2e-shared-b-w0-ae425f-serving-cert" not found Oct 30 19:32:17.974953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.974850 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbtf\" (UniqueName: \"kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.974953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.974922 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.975489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.975453 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.978159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.978132 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:17.986984 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:17.986957 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbtf\" (UniqueName: \"kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:18.489766 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:18.489716 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:18.512794 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:18.512763 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:18.727350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:18.727315 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:19.022197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:19.022160 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg"] Oct 30 19:32:19.022545 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:32:19.022240 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31558791_85f3_4319_be66_e9c51095b1e3.slice/crio-8b869c3404a480e783d1f9aaf8148b8e3cc62f397c27a90abe396549c417c9da WatchSource:0}: Error finding container 8b869c3404a480e783d1f9aaf8148b8e3cc62f397c27a90abe396549c417c9da: Status 404 returned error can't find the container with id 8b869c3404a480e783d1f9aaf8148b8e3cc62f397c27a90abe396549c417c9da Oct 30 19:32:19.858552 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:19.858511 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" event={"ID":"31558791-85f3-4319-be66-e9c51095b1e3","Type":"ContainerStarted","Data":"378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8"} Oct 30 19:32:19.858722 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:19.858576 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" event={"ID":"31558791-85f3-4319-be66-e9c51095b1e3","Type":"ContainerStarted","Data":"8b869c3404a480e783d1f9aaf8148b8e3cc62f397c27a90abe396549c417c9da"} Oct 30 19:32:19.858784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:19.858717 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:19.879466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:19.879404 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" podStartSLOduration=2.434255197 podStartE2EDuration="2.879390212s" totalImagesPullingTime="445.135015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:19.877252865 +0000 UTC m=+913.900856764" watchObservedRunningTime="2026-10-30 19:32:19.879390212 +0000 UTC m=+913.902994111" Oct 30 19:32:22.846761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:22.846725 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:32:24.380496 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:24.380458 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:32:24.488472 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.488402 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s"] Oct 30 19:32:24.493100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.493077 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.507094 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.507072 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s"] Oct 30 19:32:24.586873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.586838 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.587086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.586895 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.587086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.586992 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.587210 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.587106 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.587263 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.587214 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.587330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.587265 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.587500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.587482 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsvh\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.688759 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.688664 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsvh\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.688759 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.688718 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.689012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.688760 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.689012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.688794 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.689012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.688863 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.689012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.688942 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.689234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.689077 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.689891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.689861 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.691221 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.691193 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.691409 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.691380 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.691627 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.691607 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.691699 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.691670 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.698205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.698176 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.698510 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.698493 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsvh\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p5c6s\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:24.808689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:24.808658 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:25.097254 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:32:25.097221 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c16b2c4_cf7e_4d70_9ee8_f010bd60eb0a.slice/crio-ae54203d29ebf434a6ecfa3984628dc4675013146226d1d6bf864d02f5285aaf WatchSource:0}: Error finding container ae54203d29ebf434a6ecfa3984628dc4675013146226d1d6bf864d02f5285aaf: Status 404 returned error can't find the container with id ae54203d29ebf434a6ecfa3984628dc4675013146226d1d6bf864d02f5285aaf Oct 30 19:32:25.097571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.097545 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s"] Oct 30 19:32:25.099872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.099829 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:32:25.099949 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.099899 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:32:25.876745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.876709 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:32:25.931556 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.931469 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" event={"ID":"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a","Type":"ContainerStarted","Data":"e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b"} Oct 30 19:32:25.937553 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.937518 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" event={"ID":"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a","Type":"ContainerStarted","Data":"ae54203d29ebf434a6ecfa3984628dc4675013146226d1d6bf864d02f5285aaf"} Oct 30 19:32:25.937760 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.937746 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:25.966737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:25.966683 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" podStartSLOduration=1.966668781 podStartE2EDuration="1.966668781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:25.963259943 +0000 UTC m=+919.986863843" watchObservedRunningTime="2026-10-30 19:32:25.966668781 +0000 UTC m=+919.990272680" Oct 30 19:32:26.946911 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:26.946868 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:32:30.194585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.194544 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:30.195128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.194623 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:30.235851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.235804 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m"] Oct 30 19:32:30.244016 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.243988 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.296091 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.296050 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m"] Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.412894 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413069 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413125 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413223 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvp4s\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413266 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413316 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413341 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413405 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.414440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.413459 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515296 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515376 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515337 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515533 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515666 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515733 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515758 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515832 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvp4s\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.515909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515900 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.516020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515946 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.516020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515933 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.516020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515955 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.516020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.515970 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.516464 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.516439 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.516891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.516858 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.548491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.542929 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.552450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.550517 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvp4s\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.567446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.564699 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.595577 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.595548 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-zl27m\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:30.887924 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:30.887869 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:31.273962 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:32:31.273926 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625279b6_91d4_4ccf_805d_5d8e74d20500.slice/crio-7eddddbc663ea8db620d26c378bd82223efd682f350f4b8cdadd289cca6857e9 WatchSource:0}: Error finding container 7eddddbc663ea8db620d26c378bd82223efd682f350f4b8cdadd289cca6857e9: Status 404 returned error can't find the container with id 7eddddbc663ea8db620d26c378bd82223efd682f350f4b8cdadd289cca6857e9 Oct 30 19:32:31.276679 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:31.276651 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m"] Oct 30 19:32:31.279441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:31.278506 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:32:31.279441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:31.278668 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:32:31.279441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:31.278769 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:32:32.021377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:32.021339 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" event={"ID":"625279b6-91d4-4ccf-805d-5d8e74d20500","Type":"ContainerStarted","Data":"c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3"} Oct 30 19:32:32.022001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:32.021395 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" event={"ID":"625279b6-91d4-4ccf-805d-5d8e74d20500","Type":"ContainerStarted","Data":"7eddddbc663ea8db620d26c378bd82223efd682f350f4b8cdadd289cca6857e9"} Oct 30 19:32:32.048323 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:32.048238 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podStartSLOduration=2.048222497 podStartE2EDuration="2.048222497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:32.046701258 +0000 UTC m=+926.070305154" watchObservedRunningTime="2026-10-30 19:32:32.048222497 +0000 UTC m=+926.071826395" Oct 30 19:32:32.888877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:32.888844 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:32.890180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:32.890154 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:32.890281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:32.890209 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:33.888921 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:33.888881 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:33.889309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:33.888935 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:34.889487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:34.889449 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:34.889940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:34.889517 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:35.888513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:35.888479 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:35.888694 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:35.888534 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:36.888884 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:36.888846 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:36.889390 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:36.888941 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:37.889332 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:37.889293 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:37.889801 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:37.889366 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:38.889482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:38.889444 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-zl27m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:32:38.889881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:38.889525 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:32:40.212725 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:40.212689 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:40.213993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:40.213975 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:40.214838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:40.214819 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:32:54.463082 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:32:54.462994 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:32:55.035441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:55.030597 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:55.035441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:32:55.030726 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:24.386362 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:33:24.386301 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:33:54.404728 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:33:54.404635 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:34:15.986855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:15.986816 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t"] Oct 30 19:34:15.992169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:15.992147 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.006199 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.006172 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ca9d14-serving-cert\"" Oct 30 19:34:16.012220 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.012187 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t"] Oct 30 19:34:16.052282 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.052250 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t4z\" (UniqueName: \"kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.052445 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.052304 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.052521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.052489 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.052625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.052605 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.156240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.156202 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.156456 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.156334 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t4z\" (UniqueName: \"kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.156456 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.156381 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.156592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.156525 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.156806 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:34:16.156780 6511 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-serving-cert: secret "maas-api-e2e-shared-b-w2-ca9d14-serving-cert" not found Oct 30 19:34:16.156916 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:34:16.156858 6511 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls podName:ccc436ab-c528-448a-8e43-183b46fffc3d nodeName:}" failed. No retries permitted until 2026-10-30 19:34:16.656837674 +0000 UTC m=+1030.680441568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" (UID: "ccc436ab-c528-448a-8e43-183b46fffc3d") : secret "maas-api-e2e-shared-b-w2-ca9d14-serving-cert" not found Oct 30 19:34:16.157428 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.157387 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.164958 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.164936 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.177873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.177842 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t4z\" (UniqueName: \"kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.666461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.666402 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.677318 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.677283 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:16.916023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:16.915987 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:17.211767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:17.211738 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t"] Oct 30 19:34:18.139543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:18.139487 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" event={"ID":"ccc436ab-c528-448a-8e43-183b46fffc3d","Type":"ContainerStarted","Data":"3533be782ffc598d71b5679c375d55e6db43ff31ff411a34856e88af25c52353"} Oct 30 19:34:18.139543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:18.139561 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" event={"ID":"ccc436ab-c528-448a-8e43-183b46fffc3d","Type":"ContainerStarted","Data":"6ad0fa02c44106291b17158897c98f2a216f1e5684fb2ad6112fdcc6577074b3"} Oct 30 19:34:18.139858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:18.139592 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:18.161980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:18.161883 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" podStartSLOduration=2.673075418 podStartE2EDuration="3.161867941s" totalImagesPullingTime="488.792523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:18.159093307 +0000 UTC m=+1032.182697206" watchObservedRunningTime="2026-10-30 19:34:18.161867941 +0000 UTC m=+1032.185471839" Oct 30 19:34:23.583401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:23.583352 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s"] Oct 30 19:34:23.584211 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:23.583616 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" podUID="867374a3-3a43-411e-9f98-12ae05c958d4" containerName="maas-api" containerID="cri-o://8c4577b0312e2373b29eebd628d3e5fe27dda56571196763470b59699e4e2d3a" gracePeriod=30 Oct 30 19:34:24.173872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.173842 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:34:24.187095 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.187067 6511 generic.go:356] "Generic (PLEG): container finished" podID="867374a3-3a43-411e-9f98-12ae05c958d4" containerID="8c4577b0312e2373b29eebd628d3e5fe27dda56571196763470b59699e4e2d3a" exitCode=0 Oct 30 19:34:24.187229 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.187146 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" event={"ID":"867374a3-3a43-411e-9f98-12ae05c958d4","Type":"ContainerDied","Data":"8c4577b0312e2373b29eebd628d3e5fe27dda56571196763470b59699e4e2d3a"} Oct 30 19:34:24.187229 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.187196 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" event={"ID":"867374a3-3a43-411e-9f98-12ae05c958d4","Type":"ContainerDied","Data":"45049bc73b369dcfc5acde28549805ec8dbb1e4e6e02a2e67750dbf81ce2c501"} Oct 30 19:34:24.187229 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.187216 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45049bc73b369dcfc5acde28549805ec8dbb1e4e6e02a2e67750dbf81ce2c501" Oct 30 19:34:24.212103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.212072 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:34:24.229241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.229155 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca" pod "867374a3-3a43-411e-9f98-12ae05c958d4" (UID: "867374a3-3a43-411e-9f98-12ae05c958d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:24.231674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.231642 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca\") pod \"867374a3-3a43-411e-9f98-12ae05c958d4\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " Oct 30 19:34:24.231923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.231901 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs\") pod \"867374a3-3a43-411e-9f98-12ae05c958d4\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " Oct 30 19:34:24.232020 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.232004 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\") pod \"867374a3-3a43-411e-9f98-12ae05c958d4\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " Oct 30 19:34:24.232103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.232087 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb\" (UniqueName: \"kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb\") pod \"867374a3-3a43-411e-9f98-12ae05c958d4\" (UID: \"867374a3-3a43-411e-9f98-12ae05c958d4\") " Oct 30 19:34:24.254483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.244956 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs" pod "867374a3-3a43-411e-9f98-12ae05c958d4" (UID: "867374a3-3a43-411e-9f98-12ae05c958d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:24.254483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.247920 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls" pod "867374a3-3a43-411e-9f98-12ae05c958d4" (UID: "867374a3-3a43-411e-9f98-12ae05c958d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:24.269473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.264042 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb" pod "867374a3-3a43-411e-9f98-12ae05c958d4" (UID: "867374a3-3a43-411e-9f98-12ae05c958d4"). InnerVolumeSpecName "kube-api-access-r8vlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:24.334982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.334952 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:24.334982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.334977 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/867374a3-3a43-411e-9f98-12ae05c958d4-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:24.334982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.334988 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vlb\" (UniqueName: \"kubernetes.io/projected/867374a3-3a43-411e-9f98-12ae05c958d4-kube-api-access-r8vlb\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:24.335204 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:24.334997 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/867374a3-3a43-411e-9f98-12ae05c958d4-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:24.400775 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:34:24.400732 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:34:25.196299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:25.196261 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s" Oct 30 19:34:25.224380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:25.224353 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s"] Oct 30 19:34:25.234449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:25.234399 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2fd9a937-5487c6f49c-czm5s"] Oct 30 19:34:26.098483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:26.098454 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867374a3-3a43-411e-9f98-12ae05c958d4" path="/var/lib/kubelet/pods/867374a3-3a43-411e-9f98-12ae05c958d4/volumes" Oct 30 19:34:35.685667 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.685397 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs"] Oct 30 19:34:35.686552 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.686531 6511 cpu_manager.go:414] "containerMap: removing container" podUID="867374a3-3a43-411e-9f98-12ae05c958d4" containerName="maas-api" Oct 30 19:34:35.686552 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.686553 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867374a3-3a43-411e-9f98-12ae05c958d4" containerName="maas-api" Oct 30 19:34:35.686755 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.686737 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="867374a3-3a43-411e-9f98-12ae05c958d4" containerName="maas-api" Oct 30 19:34:35.690829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.690809 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.710012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.709960 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs"] Oct 30 19:34:35.716668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.715752 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c7880b-openshift-default-dockercfg-qzwzp\"" Oct 30 19:34:35.757260 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757224 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757497 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757626 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757544 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757626 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757580 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757626 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757608 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757792 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757673 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757792 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757717 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757792 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757740 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvhk\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.757945 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.757848 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.859774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859735 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.859774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859788 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.859774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859820 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859849 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859920 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859961 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.859984 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvhk\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860087 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860223 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860305 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860346 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860667 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860621 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860667 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860674 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.860941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.860918 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.862478 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.862407 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.863035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.863014 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.874744 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.874497 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:35.875385 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:35.875334 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvhk\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk\") pod \"e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:36.034106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:36.033720 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:36.492942 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:36.492904 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs"] Oct 30 19:34:36.493142 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:34:36.492949 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334d6da2_cbc2_40ba_ae20_ea11b6a4786a.slice/crio-b6e082c3d1d26b45cb63e5aad6aad1b82b6ec9fb5a4b61a28531b85402e491ee WatchSource:0}: Error finding container b6e082c3d1d26b45cb63e5aad6aad1b82b6ec9fb5a4b61a28531b85402e491ee: Status 404 returned error can't find the container with id b6e082c3d1d26b45cb63e5aad6aad1b82b6ec9fb5a4b61a28531b85402e491ee Oct 30 19:34:36.496091 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:36.496057 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:34:36.496197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:36.496121 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:34:36.496197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:36.496151 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:34:37.461069 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:37.460601 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" event={"ID":"334d6da2-cbc2-40ba-ae20-ea11b6a4786a","Type":"ContainerStarted","Data":"e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee"} Oct 30 19:34:37.461069 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:37.460667 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" event={"ID":"334d6da2-cbc2-40ba-ae20-ea11b6a4786a","Type":"ContainerStarted","Data":"b6e082c3d1d26b45cb63e5aad6aad1b82b6ec9fb5a4b61a28531b85402e491ee"} Oct 30 19:34:38.034743 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:38.034706 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:38.040361 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:38.040337 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:38.068940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:38.068892 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podStartSLOduration=3.068877183 podStartE2EDuration="3.068877183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:37.493669079 +0000 UTC m=+1051.517272977" watchObservedRunningTime="2026-10-30 19:34:38.068877183 +0000 UTC m=+1052.092481082" Oct 30 19:34:38.468929 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:38.468892 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:38.470073 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:34:38.470052 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:34:54.381321 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:34:54.381275 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:35:13.039249 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:13.039207 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6"] Oct 30 19:35:13.042665 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:13.042630 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3532f2d7-tkjk6"] Oct 30 19:35:14.098997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:14.098963 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472d9709-0f07-45c6-af31-34b851c166a5" path="/var/lib/kubelet/pods/472d9709-0f07-45c6-af31-34b851c166a5/volumes" Oct 30 19:35:21.023401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:21.023319 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz"] Oct 30 19:35:21.028102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:21.028073 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2e8351-gxgvz"] Oct 30 19:35:22.098968 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:22.098930 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ddf2a9-d20e-4f0b-9e83-34edf41631b9" path="/var/lib/kubelet/pods/f2ddf2a9-d20e-4f0b-9e83-34edf41631b9/volumes" Oct 30 19:35:24.379581 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:35:24.379545 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:35:27.568501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.568463 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn"] Oct 30 19:35:27.574007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.573980 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.583567 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.583544 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-95d2ac-serving-cert\"" Oct 30 19:35:27.592999 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.592972 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn"] Oct 30 19:35:27.647262 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.647231 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.647467 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.647302 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.647647 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.647623 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77b72\" (UniqueName: \"kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.647745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.647725 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.749469 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.749431 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.749656 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.749531 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.749709 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.749695 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77b72\" (UniqueName: \"kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.749758 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.749731 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.750233 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.750208 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.752517 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.752490 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.752830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.752793 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.769644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.769618 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77b72\" (UniqueName: \"kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72\") pod \"maas-api-e2e-api-b-95d2ac-6847744894-mj8mn\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:27.890758 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:27.890390 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:28.212176 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:28.212111 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn"] Oct 30 19:35:28.217006 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:35:28.216265 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4eca654_47ed_4137_861c_82aaf870aae9.slice/crio-ef545e9db951b05fce0ff9307c2409a4087d352ff5b9cd1d5d939d0d3d2aa58b WatchSource:0}: Error finding container ef545e9db951b05fce0ff9307c2409a4087d352ff5b9cd1d5d939d0d3d2aa58b: Status 404 returned error can't find the container with id ef545e9db951b05fce0ff9307c2409a4087d352ff5b9cd1d5d939d0d3d2aa58b Oct 30 19:35:28.985080 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:28.985042 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" event={"ID":"c4eca654-47ed-4137-861c-82aaf870aae9","Type":"ContainerStarted","Data":"9d0e6368f1046b827989c3d38ee81c42ac8ffa34ca0012f2e74d3ba7888b5dfb"} Oct 30 19:35:28.985646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:28.985101 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" event={"ID":"c4eca654-47ed-4137-861c-82aaf870aae9","Type":"ContainerStarted","Data":"ef545e9db951b05fce0ff9307c2409a4087d352ff5b9cd1d5d939d0d3d2aa58b"} Oct 30 19:35:28.985646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:28.985182 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:29.004705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:29.004655 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" podStartSLOduration=1.431648718 podStartE2EDuration="2.004641588s" totalImagesPullingTime="572.99287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:29.004247566 +0000 UTC m=+1103.027851462" watchObservedRunningTime="2026-10-30 19:35:29.004641588 +0000 UTC m=+1103.028245487" Oct 30 19:35:35.000588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:35.000555 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:35:39.942442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:39.941933 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:39.942442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:39.942383 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 19:35:39.978440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:39.961486 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:39.982475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:39.978939 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.032440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.031529 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 30 19:35:40.032440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.031947 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.047439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.040501 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-frx85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:35:40.047439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.040556 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.128550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.128504 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.240495 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.221506 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 19:35:40.250438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.249732 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.303444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.303372 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.304966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.304935 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.610955 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.610325 6511 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:40.648558 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:40.648508 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:35:41.216540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.216502 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp"] Oct 30 19:35:41.229936 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.229908 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.260534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.260500 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-73f96d-openshift-default-dockercfg-8vb9t\"" Oct 30 19:35:41.268479 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.268281 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp"] Oct 30 19:35:41.315266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.315210 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364162 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364207 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364260 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qjt\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364353 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364488 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364560 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364614 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.365487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.364670 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.507690 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.502590 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.517441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.508048 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.517777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.517746 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.534576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.534542 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.534960 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.529737 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.535368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.535344 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.557453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.535608 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.561689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.561655 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.592819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.592762 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qjt\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.593159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.592351 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.593409 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.592880 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.593592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.574757 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.593717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.593403 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.594627 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.594607 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.597302 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.597252 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.603830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.603787 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qjt\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.604484 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.604407 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.605849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.605822 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo\") pod \"e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:41.877443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:41.872720 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:42.205475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.205445 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp"] Oct 30 19:35:42.205750 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:35:42.205727 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1891977a_65a1_4b46_a06a_e395fe2f7b6d.slice/crio-2236738d1699aa99ca657e248aad51c9605f9ed0bbbc06513f8977c0fb4bb653 WatchSource:0}: Error finding container 2236738d1699aa99ca657e248aad51c9605f9ed0bbbc06513f8977c0fb4bb653: Status 404 returned error can't find the container with id 2236738d1699aa99ca657e248aad51c9605f9ed0bbbc06513f8977c0fb4bb653 Oct 30 19:35:42.208657 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.208623 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:35:42.208739 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.208690 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:35:42.208778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.208748 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:35:42.402351 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.401207 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" event={"ID":"1891977a-65a1-4b46-a06a-e395fe2f7b6d","Type":"ContainerStarted","Data":"c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6"} Oct 30 19:35:42.402351 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.401281 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" event={"ID":"1891977a-65a1-4b46-a06a-e395fe2f7b6d","Type":"ContainerStarted","Data":"2236738d1699aa99ca657e248aad51c9605f9ed0bbbc06513f8977c0fb4bb653"} Oct 30 19:35:42.432205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.432146 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podStartSLOduration=1.432127334 podStartE2EDuration="1.432127334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:42.431008044 +0000 UTC m=+1116.454611944" watchObservedRunningTime="2026-10-30 19:35:42.432127334 +0000 UTC m=+1116.455731234" Oct 30 19:35:42.880163 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:42.880118 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:43.886088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:43.886054 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:44.417942 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:44.417903 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:44.419208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:44.419186 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:35:48.458852 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:48.458818 6511 generic.go:356] "Generic (PLEG): container finished" podID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerID="e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee" exitCode=0 Oct 30 19:35:48.459279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:48.458890 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" event={"ID":"334d6da2-cbc2-40ba-ae20-ea11b6a4786a","Type":"ContainerDied","Data":"e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee"} Oct 30 19:35:48.459393 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:48.459365 6511 scope.go:122] "RemoveContainer" containerID="e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee" Oct 30 19:35:48.461025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:48.460997 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:35:48.461086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:48.461064 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:35:48.461123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:48.461093 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:35:49.468784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:49.468752 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" event={"ID":"334d6da2-cbc2-40ba-ae20-ea11b6a4786a","Type":"ContainerStarted","Data":"7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532"} Oct 30 19:35:50.039739 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:50.039706 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:35:50.043473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:50.043408 6511 patch_prober.go:29] interesting pod/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:50.043637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:50.043500 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 19:35:51.039717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:51.039154 6511 patch_prober.go:29] interesting pod/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:51.039717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:51.039216 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 19:35:52.035350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:52.034879 6511 patch_prober.go:29] interesting pod/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:52.035350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:52.034942 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 19:35:53.042440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:53.039542 6511 patch_prober.go:29] interesting pod/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:53.042440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:53.039614 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 19:35:53.471553 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:53.471517 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:35:54.049333 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:54.049288 6511 patch_prober.go:29] interesting pod/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:35:54.049862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:54.049357 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 19:35:54.386863 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:35:54.386825 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:35:55.070802 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:55.070772 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:35:55.072341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:35:55.072313 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:36:24.379549 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:36:24.379452 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:36:36.726075 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:36.726040 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs"] Oct 30 19:36:36.726727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:36.726637 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" containerID="cri-o://7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532" gracePeriod=30 Oct 30 19:36:40.073356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:40.073317 6511 patch_prober.go:29] interesting pod/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:36:40.073785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:40.073386 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:36:42.084525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.084496 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:36:42.106851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.106820 6511 generic.go:356] "Generic (PLEG): container finished" podID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerID="7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532" exitCode=0 Oct 30 19:36:42.107026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.106867 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" Oct 30 19:36:42.107026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.106905 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" event={"ID":"334d6da2-cbc2-40ba-ae20-ea11b6a4786a","Type":"ContainerDied","Data":"7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532"} Oct 30 19:36:42.107026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.106961 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs" event={"ID":"334d6da2-cbc2-40ba-ae20-ea11b6a4786a","Type":"ContainerDied","Data":"b6e082c3d1d26b45cb63e5aad6aad1b82b6ec9fb5a4b61a28531b85402e491ee"} Oct 30 19:36:42.107026 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.106991 6511 scope.go:122] "RemoveContainer" containerID="7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532" Oct 30 19:36:42.122858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.122831 6511 scope.go:122] "RemoveContainer" containerID="e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee" Oct 30 19:36:42.133327 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.133309 6511 scope.go:122] "RemoveContainer" containerID="7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532" Oct 30 19:36:42.133624 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:36:42.133605 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532\": container with ID starting with 7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532 not found: ID does not exist" containerID="7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532" Oct 30 19:36:42.133703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.133633 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532"} err="failed to get container status \"7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532\": rpc error: code = NotFound desc = could not find container \"7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532\": container with ID starting with 7157936bfb16d5dbecefac943fd3a9ddadf7ad0a6a01d05a6af0697478b3c532 not found: ID does not exist" Oct 30 19:36:42.133703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.133654 6511 scope.go:122] "RemoveContainer" containerID="e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee" Oct 30 19:36:42.133935 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:36:42.133912 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee\": container with ID starting with e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee not found: ID does not exist" containerID="e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee" Oct 30 19:36:42.133984 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.133946 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee"} err="failed to get container status \"e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee\": rpc error: code = NotFound desc = could not find container \"e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee\": container with ID starting with e13e8c6f588d44dcec70103b7ec61435fc7559dd497d38cf990ae3f097925eee not found: ID does not exist" Oct 30 19:36:42.223957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.223392 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.223957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.223483 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.223957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.223596 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.223957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.223770 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.223957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.223813 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:42.223957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.223858 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224166 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224195 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224209 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224234 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224278 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224337 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224344 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token\") pod \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\" (UID: \"334d6da2-cbc2-40ba-ae20-ea11b6a4786a\") " Oct 30 19:36:42.225440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.224492 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:42.226310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.226266 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.226310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.226295 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.226310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.226311 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.226629 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.226326 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.226629 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.226346 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.235895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.235868 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:36:42.235895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.235883 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "kube-api-access-xqvhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:42.236225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.236201 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:42.236333 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.236322 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo" pod "334d6da2-cbc2-40ba-ae20-ea11b6a4786a" (UID: "334d6da2-cbc2-40ba-ae20-ea11b6a4786a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:36:42.332804 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.332770 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.332975 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.332807 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.332975 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.332824 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqvhk\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-kube-api-access-xqvhk\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.332975 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.332840 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334d6da2-cbc2-40ba-ae20-ea11b6a4786a-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:42.453278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.453251 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs"] Oct 30 19:36:42.459513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.459488 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c7880b-openshift-default-56db9c9557-9vwgs"] Oct 30 19:36:42.603766 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.603682 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r"] Oct 30 19:36:42.604652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.604625 6511 cpu_manager.go:414] "containerMap: removing container" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" Oct 30 19:36:42.627814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.604652 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" Oct 30 19:36:42.627814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.622813 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" Oct 30 19:36:42.627814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.622848 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" Oct 30 19:36:42.629288 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.629257 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.645790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.642883 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6a6348-serving-cert\"" Oct 30 19:36:42.651567 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.651458 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r"] Oct 30 19:36:42.757555 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.757522 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.757970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.757948 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.758092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.757990 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltqzd\" (UniqueName: \"kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.758092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.758034 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.859446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.859291 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.859605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.859455 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.859689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.859665 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.859752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.859705 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltqzd\" (UniqueName: \"kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.860470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.860443 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.862303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.862270 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.862643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.862624 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:42.868688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:42.868662 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltqzd\" (UniqueName: \"kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd\") pod \"maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:43.002984 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:43.002947 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:43.301689 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:43.301010 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r"] Oct 30 19:36:43.307541 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:43.307520 6511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:36:44.115213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:44.115170 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" path="/var/lib/kubelet/pods/334d6da2-cbc2-40ba-ae20-ea11b6a4786a/volumes" Oct 30 19:36:44.135680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:44.135641 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" event={"ID":"6e865720-d98b-40e3-8bbc-3b63bd04f618","Type":"ContainerStarted","Data":"09817dafc1a1c2ece03761d67e3529b0125f50e686c5a94670959553bf3ede72"} Oct 30 19:36:44.135889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:44.135701 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" event={"ID":"6e865720-d98b-40e3-8bbc-3b63bd04f618","Type":"ContainerStarted","Data":"67cba3b2bab747f6c81b4b0f0da5719483ae21439293117afd902bc7e681322a"} Oct 30 19:36:44.135889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:44.135753 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:44.160101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:44.160055 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" podStartSLOduration=1.693678331 podStartE2EDuration="2.160042076s" totalImagesPullingTime="466.363745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:44.155621569 +0000 UTC m=+1178.179225468" watchObservedRunningTime="2026-10-30 19:36:44.160042076 +0000 UTC m=+1178.183645975" Oct 30 19:36:50.159461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:36:50.159326 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:36:54.379334 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:36:54.379295 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:37:06.450436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:06.449761 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:37:06.480440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:06.476708 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:37:06.666444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:06.656569 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:07.095441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:07.095346 6511 scope.go:122] "RemoveContainer" containerID="16503a573526c15bb1b8b9bc017eb21d68d8f004591c919a50e33c792fe9abf1" Oct 30 19:37:07.262645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:07.257958 6511 scope.go:122] "RemoveContainer" containerID="41e17ab8c533059747d1bc5dbcbb1c6953716ff3f4bf023fee190b873c5e59a7" Oct 30 19:37:08.003487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:07.999618 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:08.008685 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:08.004614 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:12.832857 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:12.832817 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg"] Oct 30 19:37:12.833374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:12.833026 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" containerID="cri-o://3b5a40601d947ddb8efa4f9bfff4c2dcd786f932677c08520475743eac700dda" gracePeriod=30 Oct 30 19:37:14.862837 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:14.862373 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": read tcp 10.133.0.2:39900->10.133.0.87:15021: read: connection reset by peer" start-of-body= Oct 30 19:37:14.862837 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:14.862475 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": read tcp 10.133.0.2:39900->10.133.0.87:15021: read: connection reset by peer" Oct 30 19:37:15.040722 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:15.040637 6511 generic.go:356] "Generic (PLEG): container finished" podID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerID="c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6" exitCode=0 Oct 30 19:37:15.040868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:15.040734 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" event={"ID":"1891977a-65a1-4b46-a06a-e395fe2f7b6d","Type":"ContainerDied","Data":"c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6"} Oct 30 19:37:15.041216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:15.041191 6511 scope.go:122] "RemoveContainer" containerID="c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6" Oct 30 19:37:15.043953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:15.043917 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:15.044054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:15.044011 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:15.044097 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:15.044084 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:16.051670 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:16.051638 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" event={"ID":"1891977a-65a1-4b46-a06a-e395fe2f7b6d","Type":"ContainerStarted","Data":"4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5"} Oct 30 19:37:16.879666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:16.879626 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:37:16.879857 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:16.879802 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.879919 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:16.879864 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:37:17.880315 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:17.880279 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:37:17.883940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:17.880353 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:37:18.074462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.072502 6511 generic.go:356] "Generic (PLEG): container finished" podID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerID="3b5a40601d947ddb8efa4f9bfff4c2dcd786f932677c08520475743eac700dda" exitCode=0 Oct 30 19:37:18.074462 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.072526 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" event={"ID":"4ed52396-d268-4772-bf2c-b6cc27a0ed09","Type":"ContainerDied","Data":"3b5a40601d947ddb8efa4f9bfff4c2dcd786f932677c08520475743eac700dda"} Oct 30 19:37:18.188682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.188656 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:37:18.259361 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.259328 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.259929 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.259909 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.262643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.259825 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.262746 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.260376 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:18.262872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.262851 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.263097 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263083 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.263250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263235 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.263443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263428 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.263592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263579 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.263721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263114 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.263828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263814 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.263994 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263978 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo\") pod \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\" (UID: \"4ed52396-d268-4772-bf2c-b6cc27a0ed09\") " Oct 30 19:37:18.264449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.263988 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.264449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.264160 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.265160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.265131 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.265251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.265166 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.265251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.265184 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.265251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.265199 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.265251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.265213 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.272783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.272374 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:18.272783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.272471 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:18.272783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.272558 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "kube-api-access-jpn4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.272783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.272751 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token" pod "4ed52396-d268-4772-bf2c-b6cc27a0ed09" (UID: "4ed52396-d268-4772-bf2c-b6cc27a0ed09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:18.366811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.366774 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.366811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.366812 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.367047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.366827 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpn4j\" (UniqueName: \"kubernetes.io/projected/4ed52396-d268-4772-bf2c-b6cc27a0ed09-kube-api-access-jpn4j\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.367047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.366843 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed52396-d268-4772-bf2c-b6cc27a0ed09-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:18.880449 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.880391 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:37:18.880976 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:18.880489 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:37:19.081092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.081051 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" Oct 30 19:37:19.081275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.081100 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg" event={"ID":"4ed52396-d268-4772-bf2c-b6cc27a0ed09","Type":"ContainerDied","Data":"9f10b6d1ea2995786fe5d383e4337e1b1d29d0c858793529f359d0a3934ca006"} Oct 30 19:37:19.081275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.081160 6511 scope.go:122] "RemoveContainer" containerID="3b5a40601d947ddb8efa4f9bfff4c2dcd786f932677c08520475743eac700dda" Oct 30 19:37:19.115331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.115307 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg"] Oct 30 19:37:19.148041 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.147942 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-tmlkg"] Oct 30 19:37:19.881442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.880806 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:37:19.881442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:19.880871 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:37:20.099201 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:20.099170 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" path="/var/lib/kubelet/pods/4ed52396-d268-4772-bf2c-b6cc27a0ed09/volumes" Oct 30 19:37:20.879742 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:20.879710 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:37:20.879922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:20.879757 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:37:22.075776 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:22.075391 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:37:22.076284 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:22.075973 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:37:22.077941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:22.077553 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:37:24.289631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:24.289599 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp"] Oct 30 19:37:24.297126 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:24.297098 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8"] Oct 30 19:37:24.305839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:24.305809 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g"] Oct 30 19:37:24.380780 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:24.380746 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:37:25.785953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:25.785923 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn"] Oct 30 19:37:25.786452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:25.786215 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" podUID="c4eca654-47ed-4137-861c-82aaf870aae9" containerName="maas-api" containerID="cri-o://9d0e6368f1046b827989c3d38ee81c42ac8ffa34ca0012f2e74d3ba7888b5dfb" gracePeriod=30 Oct 30 19:37:26.372160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.372120 6511 generic.go:356] "Generic (PLEG): container finished" podID="c4eca654-47ed-4137-861c-82aaf870aae9" containerID="9d0e6368f1046b827989c3d38ee81c42ac8ffa34ca0012f2e74d3ba7888b5dfb" exitCode=0 Oct 30 19:37:26.374326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.374292 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" event={"ID":"c4eca654-47ed-4137-861c-82aaf870aae9","Type":"ContainerDied","Data":"9d0e6368f1046b827989c3d38ee81c42ac8ffa34ca0012f2e74d3ba7888b5dfb"} Oct 30 19:37:26.414568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.414542 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:37:26.542821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.542794 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs\") pod \"c4eca654-47ed-4137-861c-82aaf870aae9\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " Oct 30 19:37:26.542996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.542981 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls\") pod \"c4eca654-47ed-4137-861c-82aaf870aae9\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " Oct 30 19:37:26.543050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.543038 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca\") pod \"c4eca654-47ed-4137-861c-82aaf870aae9\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " Oct 30 19:37:26.543106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.543090 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72\" (UniqueName: \"kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72\") pod \"c4eca654-47ed-4137-861c-82aaf870aae9\" (UID: \"c4eca654-47ed-4137-861c-82aaf870aae9\") " Oct 30 19:37:26.543540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.543509 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca" pod "c4eca654-47ed-4137-861c-82aaf870aae9" (UID: "c4eca654-47ed-4137-861c-82aaf870aae9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:26.543726 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.543704 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4eca654-47ed-4137-861c-82aaf870aae9-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:26.545474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.545398 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs" pod "c4eca654-47ed-4137-861c-82aaf870aae9" (UID: "c4eca654-47ed-4137-861c-82aaf870aae9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:26.554520 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.554457 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls" pod "c4eca654-47ed-4137-861c-82aaf870aae9" (UID: "c4eca654-47ed-4137-861c-82aaf870aae9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:26.578089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.578047 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72" pod "c4eca654-47ed-4137-861c-82aaf870aae9" (UID: "c4eca654-47ed-4137-861c-82aaf870aae9"). InnerVolumeSpecName "kube-api-access-77b72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:26.645727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.645646 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77b72\" (UniqueName: \"kubernetes.io/projected/c4eca654-47ed-4137-861c-82aaf870aae9-kube-api-access-77b72\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:26.645727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.645677 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:26.645727 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:26.645687 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4eca654-47ed-4137-861c-82aaf870aae9-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:27.380807 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.380771 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" Oct 30 19:37:27.380807 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.380797 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn" event={"ID":"c4eca654-47ed-4137-861c-82aaf870aae9","Type":"ContainerDied","Data":"ef545e9db951b05fce0ff9307c2409a4087d352ff5b9cd1d5d939d0d3d2aa58b"} Oct 30 19:37:27.381326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.380860 6511 scope.go:122] "RemoveContainer" containerID="9d0e6368f1046b827989c3d38ee81c42ac8ffa34ca0012f2e74d3ba7888b5dfb" Oct 30 19:37:27.419206 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.419140 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn"] Oct 30 19:37:27.435347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.435319 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-95d2ac-6847744894-mj8mn"] Oct 30 19:37:27.726607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.726520 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8"] Oct 30 19:37:27.726764 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.726737 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" containerID="cri-o://e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82" gracePeriod=30 Oct 30 19:37:27.917760 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.917724 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp"] Oct 30 19:37:27.917953 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:27.917932 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" containerID="cri-o://3fab75fe11b7673bef46eac2b5c79fe48c13b9fa29b47330d5674ac3485df926" gracePeriod=30 Oct 30 19:37:28.059144 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.059115 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z"] Oct 30 19:37:28.059340 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.059299 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" containerID="cri-o://18e8acf39724d0239cadb25865a435a69bfdf0742a49d2446bbef3d89f0d5f8b" gracePeriod=30 Oct 30 19:37:28.072823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.072795 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9"] Oct 30 19:37:28.073237 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.073187 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" containerID="cri-o://3332dc068fd49e07e5174902be6d2105ecfb22bde047b70a1b430e7e50419bbf" gracePeriod=30 Oct 30 19:37:28.077977 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.077955 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps"] Oct 30 19:37:28.078157 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.078138 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" containerID="cri-o://5e78811d47c58637e9a90001aca721f68cdb902ef370784d8e24cfe7274de23b" gracePeriod=30 Oct 30 19:37:28.102561 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.102527 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4eca654-47ed-4137-861c-82aaf870aae9" path="/var/lib/kubelet/pods/c4eca654-47ed-4137-861c-82aaf870aae9/volumes" Oct 30 19:37:28.495813 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.495725 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg"] Oct 30 19:37:28.496265 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:28.495970 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" containerID="cri-o://8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a" gracePeriod=30 Oct 30 19:37:31.094218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094183 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl"] Oct 30 19:37:31.094809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094794 6511 cpu_manager.go:414] "containerMap: removing container" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" Oct 30 19:37:31.094856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094810 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" Oct 30 19:37:31.094856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094850 6511 cpu_manager.go:414] "containerMap: removing container" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" Oct 30 19:37:31.094856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094855 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334d6da2-cbc2-40ba-ae20-ea11b6a4786a" containerName="istio-proxy" Oct 30 19:37:31.094955 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094878 6511 cpu_manager.go:414] "containerMap: removing container" podUID="c4eca654-47ed-4137-861c-82aaf870aae9" containerName="maas-api" Oct 30 19:37:31.094955 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094883 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eca654-47ed-4137-861c-82aaf870aae9" containerName="maas-api" Oct 30 19:37:31.095015 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094973 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eca654-47ed-4137-861c-82aaf870aae9" containerName="maas-api" Oct 30 19:37:31.095015 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.094981 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed52396-d268-4772-bf2c-b6cc27a0ed09" containerName="istio-proxy" Oct 30 19:37:31.100082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.100061 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.110734 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.110709 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-20e66fa4-openshift-default-dockercfg-7rz7c\"" Oct 30 19:37:31.115063 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.115036 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl"] Oct 30 19:37:31.236529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236491 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fgg\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.236695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236634 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.236695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236681 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.236777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236702 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.236843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236826 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.236882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236853 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.236882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236868 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.237004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.236985 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.237096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.237083 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.337969 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.337937 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338118 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338016 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338118 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338066 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fgg\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338118 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338149 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338177 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338251 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338277 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338299 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338350 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338595 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338409 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338702 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338764 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.338993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.338922 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.340529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.340510 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.340891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.340874 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.349563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.349506 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.349860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.349838 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fgg\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.417214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.417188 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:31.766588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.766502 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl"] Oct 30 19:37:31.770371 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:37:31.770339 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e8936b_8e98_4a59_bc26_5d56645b3457.slice/crio-30deaa6f729cdc98d307c8442e9d27f40bf7f0815122cd52e28e03554d8988ba WatchSource:0}: Error finding container 30deaa6f729cdc98d307c8442e9d27f40bf7f0815122cd52e28e03554d8988ba: Status 404 returned error can't find the container with id 30deaa6f729cdc98d307c8442e9d27f40bf7f0815122cd52e28e03554d8988ba Oct 30 19:37:31.773714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.773676 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:31.773851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.773753 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:31.773851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:31.773786 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:32.444448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:32.431522 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" event={"ID":"03e8936b-8e98-4a59-bc26-5d56645b3457","Type":"ContainerStarted","Data":"405eaa7f910f11b9046bfd3c2628b534cbacc7b0ca3aa8fbcd7610691fec92fb"} Oct 30 19:37:32.444448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:32.431593 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" event={"ID":"03e8936b-8e98-4a59-bc26-5d56645b3457","Type":"ContainerStarted","Data":"30deaa6f729cdc98d307c8442e9d27f40bf7f0815122cd52e28e03554d8988ba"} Oct 30 19:37:32.462717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:32.462668 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podStartSLOduration=1.462650427 podStartE2EDuration="1.462650427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:32.459623882 +0000 UTC m=+1226.483227784" watchObservedRunningTime="2026-10-30 19:37:32.462650427 +0000 UTC m=+1226.486254325" Oct 30 19:37:33.384644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.384614 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:37:33.407609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407577 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.407723 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407629 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.407723 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407667 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.407723 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407692 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.407904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407793 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.407904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407810 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.408007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407911 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.408007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.407963 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.408007 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.408005 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs\") pod \"42ec9afe-6e5d-4fb3-8431-d135790c9919\" (UID: \"42ec9afe-6e5d-4fb3-8431-d135790c9919\") " Oct 30 19:37:33.408881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.408803 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.408881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.408823 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.409347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.409026 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.409347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.409159 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.412395 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.412361 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.413100 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.412934 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.414602 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.414564 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.416605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.415621 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.416842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.416812 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5" pod "42ec9afe-6e5d-4fb3-8431-d135790c9919" (UID: "42ec9afe-6e5d-4fb3-8431-d135790c9919"). InnerVolumeSpecName "kube-api-access-5w4n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.437974 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.437708 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:33.439967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.439549 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:37:33.439967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.439623 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:37:33.469364 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.469323 6511 generic.go:356] "Generic (PLEG): container finished" podID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerID="18e8acf39724d0239cadb25865a435a69bfdf0742a49d2446bbef3d89f0d5f8b" exitCode=0 Oct 30 19:37:33.469861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.469398 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" event={"ID":"84c12a0a-2a8b-4132-ac2e-11e9a18beccd","Type":"ContainerDied","Data":"18e8acf39724d0239cadb25865a435a69bfdf0742a49d2446bbef3d89f0d5f8b"} Oct 30 19:37:33.472263 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.472232 6511 generic.go:356] "Generic (PLEG): container finished" podID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerID="e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82" exitCode=0 Oct 30 19:37:33.472393 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.472354 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" event={"ID":"42ec9afe-6e5d-4fb3-8431-d135790c9919","Type":"ContainerDied","Data":"e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82"} Oct 30 19:37:33.472489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.472398 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" event={"ID":"42ec9afe-6e5d-4fb3-8431-d135790c9919","Type":"ContainerDied","Data":"2fc5c4f1107462d551394906321a16c50a7e5dd870c7faaa73879e90dc0c7bb1"} Oct 30 19:37:33.472489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.472449 6511 scope.go:122] "RemoveContainer" containerID="e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82" Oct 30 19:37:33.472633 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.472614 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8" Oct 30 19:37:33.486214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.485987 6511 generic.go:356] "Generic (PLEG): container finished" podID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerID="5e78811d47c58637e9a90001aca721f68cdb902ef370784d8e24cfe7274de23b" exitCode=0 Oct 30 19:37:33.486214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.486100 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" event={"ID":"5744deea-ed79-49ad-a2e1-7472390c0a1b","Type":"ContainerDied","Data":"5e78811d47c58637e9a90001aca721f68cdb902ef370784d8e24cfe7274de23b"} Oct 30 19:37:33.489524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.488328 6511 generic.go:356] "Generic (PLEG): container finished" podID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerID="3332dc068fd49e07e5174902be6d2105ecfb22bde047b70a1b430e7e50419bbf" exitCode=0 Oct 30 19:37:33.489524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.488427 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" event={"ID":"69ffe36d-f98a-4600-8b0e-7366f8196de9","Type":"ContainerDied","Data":"3332dc068fd49e07e5174902be6d2105ecfb22bde047b70a1b430e7e50419bbf"} Oct 30 19:37:33.489913 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.489891 6511 generic.go:356] "Generic (PLEG): container finished" podID="4f18a422-a162-471f-973c-7f3c532c977a" containerID="3fab75fe11b7673bef46eac2b5c79fe48c13b9fa29b47330d5674ac3485df926" exitCode=0 Oct 30 19:37:33.490714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.490695 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" event={"ID":"4f18a422-a162-471f-973c-7f3c532c977a","Type":"ContainerDied","Data":"3fab75fe11b7673bef46eac2b5c79fe48c13b9fa29b47330d5674ac3485df926"} Oct 30 19:37:33.502408 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.502385 6511 scope.go:122] "RemoveContainer" containerID="e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82" Oct 30 19:37:33.502765 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:33.502719 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82\": container with ID starting with e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82 not found: ID does not exist" containerID="e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82" Oct 30 19:37:33.502765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.502745 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82"} err="failed to get container status \"e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82\": rpc error: code = NotFound desc = could not find container \"e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82\": container with ID starting with e1767559645a73f0d877db92b97bcaef793dfae9863d7f3a8558c7787cf95c82 not found: ID does not exist" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509663 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509694 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509730 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509745 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509762 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509775 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42ec9afe-6e5d-4fb3-8431-d135790c9919-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509808 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509822 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42ec9afe-6e5d-4fb3-8431-d135790c9919-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.511276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.509835 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4n5\" (UniqueName: \"kubernetes.io/projected/42ec9afe-6e5d-4fb3-8431-d135790c9919-kube-api-access-5w4n5\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.537751 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.532212 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8"] Oct 30 19:37:33.554615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.554585 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5frq8"] Oct 30 19:37:33.689538 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.686729 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.713843 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714076 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714176 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714256 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714337 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714382 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714524 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714560 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714609 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714644 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data\") pod \"4f18a422-a162-471f-973c-7f3c532c977a\" (UID: \"4f18a422-a162-471f-973c-7f3c532c977a\") " Oct 30 19:37:33.714877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714704 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.715557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714898 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.715557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.714903 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.715557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.715392 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.716242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.715614 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.716242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.715911 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.716242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.715947 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.716242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.716000 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.716242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.716035 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f18a422-a162-471f-973c-7f3c532c977a-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.716242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.716086 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.755463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.754477 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:37:33.760178 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.758988 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:37:33.766775 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.761556 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.766912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.766883 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.767326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.767299 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "kube-api-access-4xtqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.767730 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.767703 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token" pod "4f18a422-a162-471f-973c-7f3c532c977a" (UID: "4f18a422-a162-471f-973c-7f3c532c977a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.848704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.848666 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.848873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.848737 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.848873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.848798 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.848873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.848825 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.848873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.848868 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.849111 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.848904 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.849396 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.849365 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.849894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.849866 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.849993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.849926 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.849993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.849965 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850130 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850003 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.850130 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850055 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.850130 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850088 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.850279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850131 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850176 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.850279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850206 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.850448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850311 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.850448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850359 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850388 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850602 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850462 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.850602 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850491 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850602 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850527 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.850753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850665 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850721 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.850861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850785 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.850861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850820 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.850861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850840 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs\") pod \"5744deea-ed79-49ad-a2e1-7472390c0a1b\" (UID: \"5744deea-ed79-49ad-a2e1-7472390c0a1b\") " Oct 30 19:37:33.851017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850866 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs\") pod \"69ffe36d-f98a-4600-8b0e-7366f8196de9\" (UID: \"69ffe36d-f98a-4600-8b0e-7366f8196de9\") " Oct 30 19:37:33.851017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850936 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.851017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.850992 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv\") pod \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\" (UID: \"84c12a0a-2a8b-4132-ac2e-11e9a18beccd\") " Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851763 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851882 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851905 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851920 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xtqv\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-kube-api-access-4xtqv\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851934 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f18a422-a162-471f-973c-7f3c532c977a-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851946 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69ffe36d-f98a-4600-8b0e-7366f8196de9-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851958 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f18a422-a162-471f-973c-7f3c532c977a-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.851971 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f18a422-a162-471f-973c-7f3c532c977a-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.852237 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.852269 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.852492 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.852875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.852744 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.853510 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.852914 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.853510 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.852948 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.853619 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.853532 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.854480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.853817 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.854480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.854044 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.854480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.854221 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.855025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.854478 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.855025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.854849 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:33.857564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.857343 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.857876 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.857700 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.857876 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.857724 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:33.858170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.858113 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.858487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.858434 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "kube-api-access-v6mtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.858585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.858554 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.858880 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.858811 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.858880 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.858849 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.861042 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.861021 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq" pod "69ffe36d-f98a-4600-8b0e-7366f8196de9" (UID: "69ffe36d-f98a-4600-8b0e-7366f8196de9"). InnerVolumeSpecName "kube-api-access-nrtrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.861155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.861133 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx" pod "5744deea-ed79-49ad-a2e1-7472390c0a1b" (UID: "5744deea-ed79-49ad-a2e1-7472390c0a1b"). InnerVolumeSpecName "kube-api-access-qskcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:33.861396 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.861371 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy" pod "84c12a0a-2a8b-4132-ac2e-11e9a18beccd" (UID: "84c12a0a-2a8b-4132-ac2e-11e9a18beccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953717 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6mtv\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-kube-api-access-v6mtv\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953755 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953772 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953790 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953804 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953819 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtrq\" (UniqueName: \"kubernetes.io/projected/69ffe36d-f98a-4600-8b0e-7366f8196de9-kube-api-access-nrtrq\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953835 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskcx\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-kube-api-access-qskcx\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953848 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953863 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953877 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953890 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953903 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953916 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953929 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953942 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953959 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953975 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5744deea-ed79-49ad-a2e1-7472390c0a1b-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.953988 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69ffe36d-f98a-4600-8b0e-7366f8196de9-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.954001 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.954014 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.954027 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.954041 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5744deea-ed79-49ad-a2e1-7472390c0a1b-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.954054 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69ffe36d-f98a-4600-8b0e-7366f8196de9-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:33.954487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:33.954069 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c12a0a-2a8b-4132-ac2e-11e9a18beccd-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.002661 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.002634 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:37:34.055455 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055386 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055498 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055534 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055711 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055639 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055755 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055740 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055767 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055792 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055871 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055859 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.055972 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.055947 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert\") pod \"24facc8d-8e74-4310-9bf7-f7d299467941\" (UID: \"24facc8d-8e74-4310-9bf7-f7d299467941\") " Oct 30 19:37:34.056777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.056752 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.056955 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.056927 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.057162 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.057141 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.057235 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.057179 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:34.057278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.057256 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.059541 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.059512 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:34.059648 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.059603 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "kube-api-access-k25r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:34.059779 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.059764 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:34.060944 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.060926 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token" pod "24facc8d-8e74-4310-9bf7-f7d299467941" (UID: "24facc8d-8e74-4310-9bf7-f7d299467941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:34.105987 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.105950 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" path="/var/lib/kubelet/pods/42ec9afe-6e5d-4fb3-8431-d135790c9919/volumes" Oct 30 19:37:34.157935 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.157914 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25r9\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-kube-api-access-k25r9\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158041 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158032 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24facc8d-8e74-4310-9bf7-f7d299467941-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158097 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158152 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24facc8d-8e74-4310-9bf7-f7d299467941-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158200 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158253 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158246 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158301 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158355 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158348 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24facc8d-8e74-4310-9bf7-f7d299467941-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.158404 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.158397 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24facc8d-8e74-4310-9bf7-f7d299467941-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:34.446824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.446794 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:37:34.447389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.447363 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:37:34.501144 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.501107 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" event={"ID":"84c12a0a-2a8b-4132-ac2e-11e9a18beccd","Type":"ContainerDied","Data":"ef35724fe37c838c9fbc7966b905d55342c7b45db8110a08816116fc237e44fc"} Oct 30 19:37:34.501718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.501695 6511 scope.go:122] "RemoveContainer" containerID="18e8acf39724d0239cadb25865a435a69bfdf0742a49d2446bbef3d89f0d5f8b" Oct 30 19:37:34.501899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.501874 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z" Oct 30 19:37:34.506609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.506586 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" Oct 30 19:37:34.507219 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.507195 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps" event={"ID":"5744deea-ed79-49ad-a2e1-7472390c0a1b","Type":"ContainerDied","Data":"93f7a7bb71ae6a9640e266862eca9dd9f8f456b66b2e5708700879bf2ff429ef"} Oct 30 19:37:34.509612 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.509590 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" Oct 30 19:37:34.509715 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.509625 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9" event={"ID":"69ffe36d-f98a-4600-8b0e-7366f8196de9","Type":"ContainerDied","Data":"285227f000ad8588a62252424700753e31f61115390ab0bd9885db0f2051e3ae"} Oct 30 19:37:34.511604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.511581 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" event={"ID":"4f18a422-a162-471f-973c-7f3c532c977a","Type":"ContainerDied","Data":"7fdeca053451ede21660503185bcd99e5252826523216059799d52ae8d19aa9b"} Oct 30 19:37:34.511709 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.511647 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp" Oct 30 19:37:34.545677 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.545303 6511 generic.go:356] "Generic (PLEG): container finished" podID="24facc8d-8e74-4310-9bf7-f7d299467941" containerID="8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a" exitCode=0 Oct 30 19:37:34.545677 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.545528 6511 scope.go:122] "RemoveContainer" containerID="5e78811d47c58637e9a90001aca721f68cdb902ef370784d8e24cfe7274de23b" Oct 30 19:37:34.545829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.545681 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" Oct 30 19:37:34.546328 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.546301 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" event={"ID":"24facc8d-8e74-4310-9bf7-f7d299467941","Type":"ContainerDied","Data":"8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a"} Oct 30 19:37:34.546433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.546356 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg" event={"ID":"24facc8d-8e74-4310-9bf7-f7d299467941","Type":"ContainerDied","Data":"f51a5de753b4f8204dfd7d1e0ec57cdaa3e3418c5c96391e89ab3cca1e7d3905"} Oct 30 19:37:34.560551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.560530 6511 scope.go:122] "RemoveContainer" containerID="3332dc068fd49e07e5174902be6d2105ecfb22bde047b70a1b430e7e50419bbf" Oct 30 19:37:34.571122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.570991 6511 scope.go:122] "RemoveContainer" containerID="3fab75fe11b7673bef46eac2b5c79fe48c13b9fa29b47330d5674ac3485df926" Oct 30 19:37:34.576476 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.576452 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9"] Oct 30 19:37:34.584464 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.584369 6511 scope.go:122] "RemoveContainer" containerID="8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a" Oct 30 19:37:34.591849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.591819 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-tclm9"] Oct 30 19:37:34.602785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.602761 6511 scope.go:122] "RemoveContainer" containerID="8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a" Oct 30 19:37:34.603170 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:34.603149 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a\": container with ID starting with 8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a not found: ID does not exist" containerID="8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a" Oct 30 19:37:34.603277 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.603181 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a"} err="failed to get container status \"8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a\": rpc error: code = NotFound desc = could not find container \"8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a\": container with ID starting with 8f977f0ca30dc9b21e929cb31fd0fa27a30e942540df4fb2b460b6ca9a83ee8a not found: ID does not exist" Oct 30 19:37:34.610008 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.609982 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg"] Oct 30 19:37:34.615560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.615501 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-t2llg"] Oct 30 19:37:34.654451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.654381 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp"] Oct 30 19:37:34.659982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.659836 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-w5hjp"] Oct 30 19:37:34.668388 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.667023 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z"] Oct 30 19:37:34.671890 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.671866 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-lkn8z"] Oct 30 19:37:34.687950 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.687923 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps"] Oct 30 19:37:34.690852 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:34.690831 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-75rps"] Oct 30 19:37:35.421439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:35.417635 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:37:35.421439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:35.417707 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:37:36.110897 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.110867 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" path="/var/lib/kubelet/pods/24facc8d-8e74-4310-9bf7-f7d299467941/volumes" Oct 30 19:37:36.111382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.111368 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f18a422-a162-471f-973c-7f3c532c977a" path="/var/lib/kubelet/pods/4f18a422-a162-471f-973c-7f3c532c977a/volumes" Oct 30 19:37:36.111842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.111820 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" path="/var/lib/kubelet/pods/5744deea-ed79-49ad-a2e1-7472390c0a1b/volumes" Oct 30 19:37:36.112564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.112549 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" path="/var/lib/kubelet/pods/69ffe36d-f98a-4600-8b0e-7366f8196de9/volumes" Oct 30 19:37:36.113045 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.113030 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" path="/var/lib/kubelet/pods/84c12a0a-2a8b-4132-ac2e-11e9a18beccd/volumes" Oct 30 19:37:36.424500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.422909 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:37:36.424500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:36.422973 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:37:37.424468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:37.421802 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:37:37.424468 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:37.421876 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:37:38.518279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:38.518244 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:38.518757 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:38.518635 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:38.519617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:38.519592 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:39.318058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.318010 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" containerID="cri-o://3c3be8b1b164d2a35a297344f9137f1555529c131353aab3ad5ca377863ee1e1" gracePeriod=45 Oct 30 19:37:39.325226 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.323627 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" containerID="cri-o://10e7f18b7a61f7a3bb7bff3370c30c3e9c835fa5bf159d722e3b22bfbd4fdbef" gracePeriod=45 Oct 30 19:37:39.328790 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.328750 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" containerID="cri-o://87a93e18ec5d15453938119ec15767812826ad8ed06c765dcf99c081c6ea5af2" gracePeriod=45 Oct 30 19:37:39.601664 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.600404 6511 generic.go:356] "Generic (PLEG): container finished" podID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerID="3c3be8b1b164d2a35a297344f9137f1555529c131353aab3ad5ca377863ee1e1" exitCode=0 Oct 30 19:37:39.601664 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.600453 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" event={"ID":"7e9bd48d-c401-4276-95a0-96c1fccff988","Type":"ContainerDied","Data":"3c3be8b1b164d2a35a297344f9137f1555529c131353aab3ad5ca377863ee1e1"} Oct 30 19:37:39.603182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.602480 6511 generic.go:356] "Generic (PLEG): container finished" podID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerID="87a93e18ec5d15453938119ec15767812826ad8ed06c765dcf99c081c6ea5af2" exitCode=0 Oct 30 19:37:39.603182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.602540 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" event={"ID":"94797435-fb10-46aa-b4ad-4c1c7deee1cb","Type":"ContainerDied","Data":"87a93e18ec5d15453938119ec15767812826ad8ed06c765dcf99c081c6ea5af2"} Oct 30 19:37:39.604968 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.604540 6511 generic.go:356] "Generic (PLEG): container finished" podID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerID="10e7f18b7a61f7a3bb7bff3370c30c3e9c835fa5bf159d722e3b22bfbd4fdbef" exitCode=0 Oct 30 19:37:39.605585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.605341 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" event={"ID":"def9d7d7-d76b-4f34-89b9-b268b43454c6","Type":"ContainerDied","Data":"10e7f18b7a61f7a3bb7bff3370c30c3e9c835fa5bf159d722e3b22bfbd4fdbef"} Oct 30 19:37:39.694755 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.694367 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:37:39.699581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.699560 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:37:39.734257 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.734233 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:37:39.853254 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853163 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm\") pod \"7e9bd48d-c401-4276-95a0-96c1fccff988\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " Oct 30 19:37:39.853254 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853214 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs\") pod \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " Oct 30 19:37:39.853515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853259 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache\") pod \"7e9bd48d-c401-4276-95a0-96c1fccff988\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " Oct 30 19:37:39.853515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853314 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache\") pod \"def9d7d7-d76b-4f34-89b9-b268b43454c6\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " Oct 30 19:37:39.853515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853395 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir\") pod \"def9d7d7-d76b-4f34-89b9-b268b43454c6\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " Oct 30 19:37:39.853515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853460 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs\") pod \"7e9bd48d-c401-4276-95a0-96c1fccff988\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " Oct 30 19:37:39.853515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853496 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache\") pod \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " Oct 30 19:37:39.853788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853552 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm\") pod \"def9d7d7-d76b-4f34-89b9-b268b43454c6\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " Oct 30 19:37:39.853788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853583 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6\" (UniqueName: \"kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6\") pod \"7e9bd48d-c401-4276-95a0-96c1fccff988\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " Oct 30 19:37:39.853788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853612 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm\") pod \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " Oct 30 19:37:39.853788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853670 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7\" (UniqueName: \"kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7\") pod \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " Oct 30 19:37:39.853788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853720 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home\") pod \"7e9bd48d-c401-4276-95a0-96c1fccff988\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " Oct 30 19:37:39.853788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853762 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home\") pod \"def9d7d7-d76b-4f34-89b9-b268b43454c6\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " Oct 30 19:37:39.854082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853850 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home\") pod \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " Oct 30 19:37:39.854082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.853886 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir\") pod \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\" (UID: \"94797435-fb10-46aa-b4ad-4c1c7deee1cb\") " Oct 30 19:37:39.854082 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.854059 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh\" (UniqueName: \"kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh\") pod \"def9d7d7-d76b-4f34-89b9-b268b43454c6\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " Oct 30 19:37:39.854239 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.854150 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir\") pod \"7e9bd48d-c401-4276-95a0-96c1fccff988\" (UID: \"7e9bd48d-c401-4276-95a0-96c1fccff988\") " Oct 30 19:37:39.854239 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.854202 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs\") pod \"def9d7d7-d76b-4f34-89b9-b268b43454c6\" (UID: \"def9d7d7-d76b-4f34-89b9-b268b43454c6\") " Oct 30 19:37:39.855620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.855323 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache" pod "def9d7d7-d76b-4f34-89b9-b268b43454c6" (UID: "def9d7d7-d76b-4f34-89b9-b268b43454c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.856786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.856635 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home" pod "94797435-fb10-46aa-b4ad-4c1c7deee1cb" (UID: "94797435-fb10-46aa-b4ad-4c1c7deee1cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.857525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.857314 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir" pod "94797435-fb10-46aa-b4ad-4c1c7deee1cb" (UID: "94797435-fb10-46aa-b4ad-4c1c7deee1cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.857525 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.857330 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6" pod "7e9bd48d-c401-4276-95a0-96c1fccff988" (UID: "7e9bd48d-c401-4276-95a0-96c1fccff988"). InnerVolumeSpecName "kube-api-access-5mts6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:39.857715 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.857575 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs" pod "def9d7d7-d76b-4f34-89b9-b268b43454c6" (UID: "def9d7d7-d76b-4f34-89b9-b268b43454c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:39.857990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.857789 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir" pod "7e9bd48d-c401-4276-95a0-96c1fccff988" (UID: "7e9bd48d-c401-4276-95a0-96c1fccff988"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.857990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.857839 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache" pod "94797435-fb10-46aa-b4ad-4c1c7deee1cb" (UID: "94797435-fb10-46aa-b4ad-4c1c7deee1cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.858153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.858018 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache" pod "7e9bd48d-c401-4276-95a0-96c1fccff988" (UID: "7e9bd48d-c401-4276-95a0-96c1fccff988"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.858385 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.858354 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir" pod "def9d7d7-d76b-4f34-89b9-b268b43454c6" (UID: "def9d7d7-d76b-4f34-89b9-b268b43454c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.858906 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.858886 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home" pod "def9d7d7-d76b-4f34-89b9-b268b43454c6" (UID: "def9d7d7-d76b-4f34-89b9-b268b43454c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.859683 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.859636 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm" pod "def9d7d7-d76b-4f34-89b9-b268b43454c6" (UID: "def9d7d7-d76b-4f34-89b9-b268b43454c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.860825 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.860794 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7" pod "94797435-fb10-46aa-b4ad-4c1c7deee1cb" (UID: "94797435-fb10-46aa-b4ad-4c1c7deee1cb"). InnerVolumeSpecName "kube-api-access-hcwl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:39.860933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.860829 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm" pod "94797435-fb10-46aa-b4ad-4c1c7deee1cb" (UID: "94797435-fb10-46aa-b4ad-4c1c7deee1cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.862962 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.862916 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home" pod "7e9bd48d-c401-4276-95a0-96c1fccff988" (UID: "7e9bd48d-c401-4276-95a0-96c1fccff988"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.863922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.863896 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm" pod "7e9bd48d-c401-4276-95a0-96c1fccff988" (UID: "7e9bd48d-c401-4276-95a0-96c1fccff988"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:39.864125 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.864095 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs" pod "7e9bd48d-c401-4276-95a0-96c1fccff988" (UID: "7e9bd48d-c401-4276-95a0-96c1fccff988"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:39.864295 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.864270 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh" pod "def9d7d7-d76b-4f34-89b9-b268b43454c6" (UID: "def9d7d7-d76b-4f34-89b9-b268b43454c6"). InnerVolumeSpecName "kube-api-access-dffgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:39.865233 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.865207 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs" pod "94797435-fb10-46aa-b4ad-4c1c7deee1cb" (UID: "94797435-fb10-46aa-b4ad-4c1c7deee1cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:39.957213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957166 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957206 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957224 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dffgh\" (UniqueName: \"kubernetes.io/projected/def9d7d7-d76b-4f34-89b9-b268b43454c6-kube-api-access-dffgh\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957242 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957258 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def9d7d7-d76b-4f34-89b9-b268b43454c6-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957269 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957282 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94797435-fb10-46aa-b4ad-4c1c7deee1cb-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957294 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957308 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957322 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957336 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9bd48d-c401-4276-95a0-96c1fccff988-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957351 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957364 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957378 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mts6\" (UniqueName: \"kubernetes.io/projected/7e9bd48d-c401-4276-95a0-96c1fccff988-kube-api-access-5mts6\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957395 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94797435-fb10-46aa-b4ad-4c1c7deee1cb-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957451 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwl7\" (UniqueName: \"kubernetes.io/projected/94797435-fb10-46aa-b4ad-4c1c7deee1cb-kube-api-access-hcwl7\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957469 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9bd48d-c401-4276-95a0-96c1fccff988-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:39.957578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:39.957482 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def9d7d7-d76b-4f34-89b9-b268b43454c6-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:40.614212 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.614180 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" Oct 30 19:37:40.614719 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.614231 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp" event={"ID":"7e9bd48d-c401-4276-95a0-96c1fccff988","Type":"ContainerDied","Data":"a40670cf2b1f6d3a6dd1986e4ec16fefa900726a7f7faadba9c6b0095dde3aa6"} Oct 30 19:37:40.614719 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.614289 6511 scope.go:122] "RemoveContainer" containerID="3c3be8b1b164d2a35a297344f9137f1555529c131353aab3ad5ca377863ee1e1" Oct 30 19:37:40.615768 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.615753 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" Oct 30 19:37:40.615845 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.615793 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" event={"ID":"94797435-fb10-46aa-b4ad-4c1c7deee1cb","Type":"ContainerDied","Data":"762afe2a5a8503b5550f26da0322f7672ebc0bb56395d4a640c9260f6e18d02d"} Oct 30 19:37:40.617821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.617797 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" event={"ID":"def9d7d7-d76b-4f34-89b9-b268b43454c6","Type":"ContainerDied","Data":"2830d23748f6fcb675f7f3299eb8f2353dbb0236d185e363723da04a3041607f"} Oct 30 19:37:40.617930 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.617833 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8" Oct 30 19:37:40.625165 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.625149 6511 scope.go:122] "RemoveContainer" containerID="87a93e18ec5d15453938119ec15767812826ad8ed06c765dcf99c081c6ea5af2" Oct 30 19:37:40.633768 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.633752 6511 scope.go:122] "RemoveContainer" containerID="10e7f18b7a61f7a3bb7bff3370c30c3e9c835fa5bf159d722e3b22bfbd4fdbef" Oct 30 19:37:40.645967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.645944 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:40.660296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.660272 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp"] Oct 30 19:37:40.674834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.674813 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-2-w5-60577a-kserve-cc79b8569-n5fmp"] Oct 30 19:37:40.697947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.697921 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8"] Oct 30 19:37:40.707940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.707917 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/distinct-w5-60577a-kserve-577c989554-wnjq8"] Oct 30 19:37:40.726466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.726444 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g"] Oct 30 19:37:40.737001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:40.736981 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-60577a/premium-w5-60577a-kserve-6ff5bddc9f-gwg6g"] Oct 30 19:37:42.098342 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:42.098312 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" path="/var/lib/kubelet/pods/7e9bd48d-c401-4276-95a0-96c1fccff988/volumes" Oct 30 19:37:42.098834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:42.098818 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" path="/var/lib/kubelet/pods/94797435-fb10-46aa-b4ad-4c1c7deee1cb/volumes" Oct 30 19:37:42.099208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:42.099193 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" path="/var/lib/kubelet/pods/def9d7d7-d76b-4f34-89b9-b268b43454c6/volumes" Oct 30 19:37:46.144586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.144550 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq"] Oct 30 19:37:46.145458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145435 6511 cpu_manager.go:414] "containerMap: removing container" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145459 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145490 6511 cpu_manager.go:414] "containerMap: removing container" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145497 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145521 6511 cpu_manager.go:414] "containerMap: removing container" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145528 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145551 6511 cpu_manager.go:414] "containerMap: removing container" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145557 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145569 6511 cpu_manager.go:414] "containerMap: removing container" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" Oct 30 19:37:46.145575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145576 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145587 6511 cpu_manager.go:414] "containerMap: removing container" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145594 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145632 6511 cpu_manager.go:414] "containerMap: removing container" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145639 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145660 6511 cpu_manager.go:414] "containerMap: removing container" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145667 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145677 6511 cpu_manager.go:414] "containerMap: removing container" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145683 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145823 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9bd48d-c401-4276-95a0-96c1fccff988" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145841 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="def9d7d7-d76b-4f34-89b9-b268b43454c6" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145852 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f18a422-a162-471f-973c-7f3c532c977a" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145861 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c12a0a-2a8b-4132-ac2e-11e9a18beccd" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145874 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="94797435-fb10-46aa-b4ad-4c1c7deee1cb" containerName="main" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145885 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ffe36d-f98a-4600-8b0e-7366f8196de9" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145896 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5744deea-ed79-49ad-a2e1-7472390c0a1b" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145906 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="24facc8d-8e74-4310-9bf7-f7d299467941" containerName="istio-proxy" Oct 30 19:37:46.146023 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.145914 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ec9afe-6e5d-4fb3-8431-d135790c9919" containerName="istio-proxy" Oct 30 19:37:46.152718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.152698 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.164907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.164881 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq"] Oct 30 19:37:46.193489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.193458 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvfpx\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.193730 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.193517 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.193730 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.193613 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.193730 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.193667 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.193962 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.193804 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.194035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.194002 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.194115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.194099 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.194203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.194186 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.194338 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.194321 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296095 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296141 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296198 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296229 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvfpx\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296290 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296263 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296301 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296338 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296405 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296552 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296632 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.296927 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296897 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.297104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.296952 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.297166 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.297125 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.297232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.297207 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.298654 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.298629 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.298915 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.298898 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.305486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.305462 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.305827 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.305808 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvfpx\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx\") pod \"e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.477576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.477500 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:46.769220 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:37:46.769189 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea9ae7a_2985_4e31_9bf0_e0d4dcfd5d7b.slice/crio-eb2df7982fb71005efe9ea355b085b22f042db338e4d4a6319195d64928907f9 WatchSource:0}: Error finding container eb2df7982fb71005efe9ea355b085b22f042db338e4d4a6319195d64928907f9: Status 404 returned error can't find the container with id eb2df7982fb71005efe9ea355b085b22f042db338e4d4a6319195d64928907f9 Oct 30 19:37:46.769904 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.769809 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq"] Oct 30 19:37:46.772694 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.772655 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:46.772788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.772740 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:46.772837 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:46.772787 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:37:47.678509 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:47.678468 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" event={"ID":"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b","Type":"ContainerStarted","Data":"d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38"} Oct 30 19:37:47.678951 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:47.678571 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" event={"ID":"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b","Type":"ContainerStarted","Data":"eb2df7982fb71005efe9ea355b085b22f042db338e4d4a6319195d64928907f9"} Oct 30 19:37:47.706378 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:47.706311 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podStartSLOduration=1.706290114 podStartE2EDuration="1.706290114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:47.701329665 +0000 UTC m=+1241.724933587" watchObservedRunningTime="2026-10-30 19:37:47.706290114 +0000 UTC m=+1241.729894015" Oct 30 19:37:48.478446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:48.478002 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:48.485736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:48.485613 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:37:48.485967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:48.485829 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 19:37:49.478203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:49.478170 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:37:49.478625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:49.478232 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 19:37:50.484440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:50.481017 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:37:50.484440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:50.481105 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 19:37:51.479660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:51.478108 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:37:51.479660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:51.478180 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 19:37:51.874324 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:51.874285 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp"] Oct 30 19:37:51.874799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:51.874608 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" containerID="cri-o://4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5" gracePeriod=30 Oct 30 19:37:52.113581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.113541 6511 patch_prober.go:29] interesting pod/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:37:52.113881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.113850 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:37:52.478057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.477988 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:37:52.478237 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.478074 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 19:37:52.597442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.596770 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq"] Oct 30 19:37:52.597442 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.597004 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" containerID="cri-o://d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38" gracePeriod=30 Oct 30 19:37:52.609829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.609796 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl"] Oct 30 19:37:52.610048 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:52.610024 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" containerID="cri-o://405eaa7f910f11b9046bfd3c2628b534cbacc7b0ca3aa8fbcd7610691fec92fb" gracePeriod=30 Oct 30 19:37:53.521140 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:53.521105 6511 patch_prober.go:29] interesting pod/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:37:53.521560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:53.521165 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:37:54.394438 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:54.388997 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:37:57.179579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.179499 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:37:57.201546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201519 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.201690 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201655 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.201753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201737 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.201794 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201781 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.201894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201873 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.202065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201971 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:57.202065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.201986 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.202065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202048 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.202065 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202052 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:57.202299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202094 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.202299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202124 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert\") pod \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\" (UID: \"1891977a-65a1-4b46-a06a-e395fe2f7b6d\") " Oct 30 19:37:57.202299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202196 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:57.202517 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202348 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:57.203060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.202920 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:57.203060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.203001 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.203060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.203026 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.203060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.203041 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.203060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.203055 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.204466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.204402 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:57.204564 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.204465 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:57.204908 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.204894 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:57.205193 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.205170 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt" pod "1891977a-65a1-4b46-a06a-e395fe2f7b6d" (UID: "1891977a-65a1-4b46-a06a-e395fe2f7b6d"). InnerVolumeSpecName "kube-api-access-m4qjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:57.304649 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.304610 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.304809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.304659 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.304809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.304676 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.304809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.304691 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qjt\" (UniqueName: \"kubernetes.io/projected/1891977a-65a1-4b46-a06a-e395fe2f7b6d-kube-api-access-m4qjt\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.304809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.304708 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1891977a-65a1-4b46-a06a-e395fe2f7b6d-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:57.774918 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.774883 6511 generic.go:356] "Generic (PLEG): container finished" podID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerID="4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5" exitCode=0 Oct 30 19:37:57.775048 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.774938 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" Oct 30 19:37:57.775048 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.774987 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" event={"ID":"1891977a-65a1-4b46-a06a-e395fe2f7b6d","Type":"ContainerDied","Data":"4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5"} Oct 30 19:37:57.775048 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.775034 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp" event={"ID":"1891977a-65a1-4b46-a06a-e395fe2f7b6d","Type":"ContainerDied","Data":"2236738d1699aa99ca657e248aad51c9605f9ed0bbbc06513f8977c0fb4bb653"} Oct 30 19:37:57.775170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.775063 6511 scope.go:122] "RemoveContainer" containerID="4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5" Oct 30 19:37:57.779390 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.778601 6511 generic.go:356] "Generic (PLEG): container finished" podID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerID="405eaa7f910f11b9046bfd3c2628b534cbacc7b0ca3aa8fbcd7610691fec92fb" exitCode=0 Oct 30 19:37:57.779390 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.778660 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" event={"ID":"03e8936b-8e98-4a59-bc26-5d56645b3457","Type":"ContainerDied","Data":"405eaa7f910f11b9046bfd3c2628b534cbacc7b0ca3aa8fbcd7610691fec92fb"} Oct 30 19:37:57.801616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.801591 6511 scope.go:122] "RemoveContainer" containerID="c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6" Oct 30 19:37:57.813311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.813280 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp"] Oct 30 19:37:57.823500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.823468 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-73f96d-openshift-default-66d877c755-9nvtp"] Oct 30 19:37:57.829806 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.829783 6511 scope.go:122] "RemoveContainer" containerID="4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5" Oct 30 19:37:57.830115 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:57.830091 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5\": container with ID starting with 4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5 not found: ID does not exist" containerID="4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5" Oct 30 19:37:57.830232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.830127 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5"} err="failed to get container status \"4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5\": rpc error: code = NotFound desc = could not find container \"4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5\": container with ID starting with 4b9d8c2e055c008d2a86f9c44fddbf68c2253c09d3cf166cb9c33177ed35f4a5 not found: ID does not exist" Oct 30 19:37:57.830232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.830148 6511 scope.go:122] "RemoveContainer" containerID="c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6" Oct 30 19:37:57.830347 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:57.830332 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6\": container with ID starting with c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6 not found: ID does not exist" containerID="c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6" Oct 30 19:37:57.830393 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.830350 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6"} err="failed to get container status \"c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6\": rpc error: code = NotFound desc = could not find container \"c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6\": container with ID starting with c13ddf6724f0da7f9ef05736e45cb9f09791d56a567a1a1fec8c5aec9361a1d6 not found: ID does not exist" Oct 30 19:37:57.989855 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:57.989828 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.038840 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.038971 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039053 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039136 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039169 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039258 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039311 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039355 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.044440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.039505 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy\") pod \"03e8936b-8e98-4a59-bc26-5d56645b3457\" (UID: \"03e8936b-8e98-4a59-bc26-5d56645b3457\") " Oct 30 19:37:58.057899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.057842 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.060650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.058555 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.060650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.058744 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.060650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.059291 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.060650 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.059672 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:58.067948 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.063289 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:58.076893 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.076849 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "kube-api-access-w9fgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.088763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.088596 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.093073 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.093031 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token" pod "03e8936b-8e98-4a59-bc26-5d56645b3457" (UID: "03e8936b-8e98-4a59-bc26-5d56645b3457"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.105304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.104587 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" path="/var/lib/kubelet/pods/1891977a-65a1-4b46-a06a-e395fe2f7b6d/volumes" Oct 30 19:37:58.142021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.141988 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03e8936b-8e98-4a59-bc26-5d56645b3457-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142020 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142037 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142053 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142069 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142081 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142093 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142107 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03e8936b-8e98-4a59-bc26-5d56645b3457-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.142245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.142120 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9fgg\" (UniqueName: \"kubernetes.io/projected/03e8936b-8e98-4a59-bc26-5d56645b3457-kube-api-access-w9fgg\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.746120 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.746081 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:58.811138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.811100 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" event={"ID":"03e8936b-8e98-4a59-bc26-5d56645b3457","Type":"ContainerDied","Data":"30deaa6f729cdc98d307c8442e9d27f40bf7f0815122cd52e28e03554d8988ba"} Oct 30 19:37:58.811319 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.811176 6511 scope.go:122] "RemoveContainer" containerID="405eaa7f910f11b9046bfd3c2628b534cbacc7b0ca3aa8fbcd7610691fec92fb" Oct 30 19:37:58.811319 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.811282 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl" Oct 30 19:37:58.817104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.817063 6511 generic.go:356] "Generic (PLEG): container finished" podID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerID="d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38" exitCode=0 Oct 30 19:37:58.817218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.817127 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" event={"ID":"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b","Type":"ContainerDied","Data":"d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38"} Oct 30 19:37:58.817218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.817171 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" event={"ID":"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b","Type":"ContainerDied","Data":"eb2df7982fb71005efe9ea355b085b22f042db338e4d4a6319195d64928907f9"} Oct 30 19:37:58.817343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.817233 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq" Oct 30 19:37:58.831101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.831060 6511 scope.go:122] "RemoveContainer" containerID="d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38" Oct 30 19:37:58.843465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.842169 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl"] Oct 30 19:37:58.857012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.856987 6511 scope.go:122] "RemoveContainer" containerID="d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38" Oct 30 19:37:58.857811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.857783 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.857938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.857840 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.857938 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:37:58.857885 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38\": container with ID starting with d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38 not found: ID does not exist" containerID="d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38" Oct 30 19:37:58.857938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.857921 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38"} err="failed to get container status \"d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38\": rpc error: code = NotFound desc = could not find container \"d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38\": container with ID starting with d511c36c0bf87a29c7f42b07c11f19f0c93ec15a0b1f087756463da509463d38 not found: ID does not exist" Oct 30 19:37:58.857938 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.857892 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.858155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.857979 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.858155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.858014 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.858155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.858119 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.858295 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.858158 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.858402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.858382 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.858658 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.858642 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo\") pod \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\" (UID: \"1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b\") " Oct 30 19:37:58.860258 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.860223 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.860654 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.860628 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-c6gjl"] Oct 30 19:37:58.861273 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.861250 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.861609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.861586 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.862101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.861910 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.862101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.862004 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:58.864737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.864700 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:37:58.865051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.865027 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.866615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.866590 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "kube-api-access-tvfpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.869663 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.869635 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo" pod "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" (UID: "1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:37:58.961241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961201 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961243 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961258 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961274 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961288 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961303 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961319 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961332 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.961573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:58.961346 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvfpx\" (UniqueName: \"kubernetes.io/projected/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b-kube-api-access-tvfpx\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:59.153635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:59.153603 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq"] Oct 30 19:37:59.156768 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:37:59.156706 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20e66fa4-openshift-default-6ff5b8c94f-vm6lq"] Oct 30 19:38:00.103137 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:00.103105 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" path="/var/lib/kubelet/pods/03e8936b-8e98-4a59-bc26-5d56645b3457/volumes" Oct 30 19:38:00.103810 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:00.103790 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" path="/var/lib/kubelet/pods/1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b/volumes" Oct 30 19:38:06.369735 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.369703 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp"] Oct 30 19:38:06.370262 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370247 6511 cpu_manager.go:414] "containerMap: removing container" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370263 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370283 6511 cpu_manager.go:414] "containerMap: removing container" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370288 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370296 6511 cpu_manager.go:414] "containerMap: removing container" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370301 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370313 6511 cpu_manager.go:414] "containerMap: removing container" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" Oct 30 19:38:06.370320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370318 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" Oct 30 19:38:06.370569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370406 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea9ae7a-2985-4e31-9bf0-e0d4dcfd5d7b" containerName="istio-proxy" Oct 30 19:38:06.370569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370433 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" Oct 30 19:38:06.370569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.370443 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e8936b-8e98-4a59-bc26-5d56645b3457" containerName="istio-proxy" Oct 30 19:38:06.375234 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.375216 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.393036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.393013 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-579ddb-serving-cert\"" Oct 30 19:38:06.401604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.401579 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp"] Oct 30 19:38:06.474043 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.474005 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.474186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.474049 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvqk6\" (UniqueName: \"kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.474246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.474233 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.474301 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.474272 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.576452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.576377 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.577353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.577325 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.577534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.577458 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.577534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.577495 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvqk6\" (UniqueName: \"kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.579081 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.579057 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.581645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.580362 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.581645 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.580878 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.592202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.592161 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvqk6\" (UniqueName: \"kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6\") pod \"maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:06.691480 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:06.691368 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:07.006856 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:07.006830 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp"] Oct 30 19:38:07.008849 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:38:07.008811 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7c687e_86c8_4047_844e_ca762470c1ba.slice/crio-c408f65484c157b13230dcb28f2b8a8b62a92d5d10027ff71c651d3dde648315 WatchSource:0}: Error finding container c408f65484c157b13230dcb28f2b8a8b62a92d5d10027ff71c651d3dde648315: Status 404 returned error can't find the container with id c408f65484c157b13230dcb28f2b8a8b62a92d5d10027ff71c651d3dde648315 Oct 30 19:38:07.910136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:07.909910 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" event={"ID":"aa7c687e-86c8-4047-844e-ca762470c1ba","Type":"ContainerStarted","Data":"2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e"} Oct 30 19:38:07.910136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:07.909973 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" event={"ID":"aa7c687e-86c8-4047-844e-ca762470c1ba","Type":"ContainerStarted","Data":"c408f65484c157b13230dcb28f2b8a8b62a92d5d10027ff71c651d3dde648315"} Oct 30 19:38:07.916728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:07.910162 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:07.946125 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:07.946051 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" podStartSLOduration=1.5688634320000001 podStartE2EDuration="1.946032199s" totalImagesPullingTime="377.168767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:07.942896047 +0000 UTC m=+1261.966499940" watchObservedRunningTime="2026-10-30 19:38:07.946032199 +0000 UTC m=+1261.969636101" Oct 30 19:38:11.637439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:11.617837 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p"] Oct 30 19:38:11.637439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:11.618096 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" containerID="cri-o://3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039" gracePeriod=30 Oct 30 19:38:11.648507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:11.646344 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79"] Oct 30 19:38:11.648507 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:11.646595 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" containerID="cri-o://9723fc2ff6fa1e7b08971e9f7ffb7f442e3cc832b80f697b89b878edaa25465b" gracePeriod=30 Oct 30 19:38:13.928658 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:13.928621 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:38:16.960967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.960944 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:38:16.982703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.982669 6511 generic.go:356] "Generic (PLEG): container finished" podID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerID="3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039" exitCode=0 Oct 30 19:38:16.982833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.982724 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" Oct 30 19:38:16.982833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.982760 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" event={"ID":"a037d7c5-52cf-495b-967b-792c42da4ce6","Type":"ContainerDied","Data":"3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039"} Oct 30 19:38:16.982833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.982819 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p" event={"ID":"a037d7c5-52cf-495b-967b-792c42da4ce6","Type":"ContainerDied","Data":"d2a2b45351f31f706645e86a0a0bd16b93c492bb865c2a0530753d90a21805c0"} Oct 30 19:38:16.983922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.982848 6511 scope.go:122] "RemoveContainer" containerID="3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039" Oct 30 19:38:16.999446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.999390 6511 generic.go:356] "Generic (PLEG): container finished" podID="db67505d-0f56-4867-9f46-481643eba36b" containerID="9723fc2ff6fa1e7b08971e9f7ffb7f442e3cc832b80f697b89b878edaa25465b" exitCode=0 Oct 30 19:38:16.999548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.999509 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" event={"ID":"db67505d-0f56-4867-9f46-481643eba36b","Type":"ContainerDied","Data":"9723fc2ff6fa1e7b08971e9f7ffb7f442e3cc832b80f697b89b878edaa25465b"} Oct 30 19:38:16.999624 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.999607 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" event={"ID":"db67505d-0f56-4867-9f46-481643eba36b","Type":"ContainerDied","Data":"196fe3c5700160d490580ae6228aa1f85601d13de355883fad9f1911e07b8e99"} Oct 30 19:38:16.999684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:16.999642 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="196fe3c5700160d490580ae6228aa1f85601d13de355883fad9f1911e07b8e99" Oct 30 19:38:17.018703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.013979 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:38:17.023844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.021989 6511 scope.go:122] "RemoveContainer" containerID="3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039" Oct 30 19:38:17.023844 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:38:17.022572 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039\": container with ID starting with 3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039 not found: ID does not exist" containerID="3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039" Oct 30 19:38:17.023844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.022623 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039"} err="failed to get container status \"3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039\": rpc error: code = NotFound desc = could not find container \"3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039\": container with ID starting with 3a6ab383e269407a2342dd6e13ad620f5e3c4db96d9e74427a5c6e25f3455039 not found: ID does not exist" Oct 30 19:38:17.047654 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047625 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.047804 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047737 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.047804 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047771 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.047883 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047802 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.047883 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047871 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.047948 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047937 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.048005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.047988 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.048076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.048062 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.048222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.048202 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket\") pod \"a037d7c5-52cf-495b-967b-792c42da4ce6\" (UID: \"a037d7c5-52cf-495b-967b-792c42da4ce6\") " Oct 30 19:38:17.048566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.048529 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.048959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.048935 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.049107 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.049082 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.049359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.049335 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:17.049466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.049402 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.049537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.049492 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.053679 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.053652 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "kube-api-access-78r4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.053832 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.053750 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:17.053946 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.053903 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.055397 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.055376 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token" pod "a037d7c5-52cf-495b-967b-792c42da4ce6" (UID: "a037d7c5-52cf-495b-967b-792c42da4ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.149682 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.149647 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.149833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.149697 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.149833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.149729 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.149833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.149807 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.149967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.149877 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.149967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.149935 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.150068 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150048 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.150122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150088 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.150227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150209 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.150299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150245 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs\") pod \"db67505d-0f56-4867-9f46-481643eba36b\" (UID: \"db67505d-0f56-4867-9f46-481643eba36b\") " Oct 30 19:38:17.150299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150207 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.150299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150247 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:17.150474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150439 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.150777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.150685 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.151307 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151282 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151314 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151333 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78r4v\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-kube-api-access-78r4v\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151350 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151365 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a037d7c5-52cf-495b-967b-792c42da4ce6-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151380 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151394 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151408 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151442 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a037d7c5-52cf-495b-967b-792c42da4ce6-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151456 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151470 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151483 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db67505d-0f56-4867-9f46-481643eba36b-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.151489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.151497 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a037d7c5-52cf-495b-967b-792c42da4ce6-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.152849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.152171 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.152849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.152335 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:38:17.152849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.152603 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "kube-api-access-dbs5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:17.152849 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.152730 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo" pod "db67505d-0f56-4867-9f46-481643eba36b" (UID: "db67505d-0f56-4867-9f46-481643eba36b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:38:17.253267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.253183 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db67505d-0f56-4867-9f46-481643eba36b-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.253267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.253214 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.253267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.253229 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbs5m\" (UniqueName: \"kubernetes.io/projected/db67505d-0f56-4867-9f46-481643eba36b-kube-api-access-dbs5m\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.253267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.253239 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db67505d-0f56-4867-9f46-481643eba36b-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:17.334635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.333440 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p"] Oct 30 19:38:17.337580 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:17.337548 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-sbl2p"] Oct 30 19:38:18.008954 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:18.008867 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79" Oct 30 19:38:18.039997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:18.039944 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79"] Oct 30 19:38:18.044940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:18.044901 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-60577a-openshift-default-7c8cb87747-m9f79"] Oct 30 19:38:18.100673 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:18.100639 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" path="/var/lib/kubelet/pods/a037d7c5-52cf-495b-967b-792c42da4ce6/volumes" Oct 30 19:38:18.101151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:18.101134 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db67505d-0f56-4867-9f46-481643eba36b" path="/var/lib/kubelet/pods/db67505d-0f56-4867-9f46-481643eba36b/volumes" Oct 30 19:38:19.582379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.582336 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9"] Oct 30 19:38:19.583106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583083 6511 cpu_manager.go:414] "containerMap: removing container" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" Oct 30 19:38:19.583202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583108 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" Oct 30 19:38:19.583202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583166 6511 cpu_manager.go:414] "containerMap: removing container" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" Oct 30 19:38:19.583202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583175 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" Oct 30 19:38:19.583359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583311 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a037d7c5-52cf-495b-967b-792c42da4ce6" containerName="istio-proxy" Oct 30 19:38:19.583359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583328 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="1891977a-65a1-4b46-a06a-e395fe2f7b6d" containerName="istio-proxy" Oct 30 19:38:19.583359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.583342 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="db67505d-0f56-4867-9f46-481643eba36b" containerName="istio-proxy" Oct 30 19:38:19.588166 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.588144 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.596574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.596555 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9"] Oct 30 19:38:19.596851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.596833 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d1d256-serving-cert\"" Oct 30 19:38:19.684989 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.684958 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.685159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.684994 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xpw\" (UniqueName: \"kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.685258 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.685173 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.685333 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.685296 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.786233 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.786197 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.786648 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.786624 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.787168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.787148 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xpw\" (UniqueName: \"kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.787670 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.787637 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.788666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.788625 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.790190 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.790164 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.791156 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.791132 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.802851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.802827 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xpw\" (UniqueName: \"kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw\") pod \"maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:19.903625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:19.903547 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:20.240392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:20.240358 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9"] Oct 30 19:38:21.038750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:21.038712 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" event={"ID":"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8","Type":"ContainerStarted","Data":"4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b"} Oct 30 19:38:21.039123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:21.038798 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" event={"ID":"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8","Type":"ContainerStarted","Data":"8447da6c2a37b8d40e21414c8857e4d30cdb8b1aef679d08188263b37e70b462"} Oct 30 19:38:21.039123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:21.038821 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:21.062340 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:21.062256 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" podStartSLOduration=1.520335864 podStartE2EDuration="2.062241767s" totalImagesPullingTime="541.905903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:21.059538128 +0000 UTC m=+1275.083142027" watchObservedRunningTime="2026-10-30 19:38:21.062241767 +0000 UTC m=+1275.085845666" Oct 30 19:38:24.379683 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:38:24.379644 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:38:27.070439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:27.065601 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:38:42.907686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:42.907640 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r"] Oct 30 19:38:42.908272 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:42.908235 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" podUID="6e865720-d98b-40e3-8bbc-3b63bd04f618" containerName="maas-api" containerID="cri-o://09817dafc1a1c2ece03761d67e3529b0125f50e686c5a94670959553bf3ede72" gracePeriod=30 Oct 30 19:38:42.924727 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:38:42.924695 6511 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e865720_d98b_40e3_8bbc_3b63bd04f618.slice/crio-09817dafc1a1c2ece03761d67e3529b0125f50e686c5a94670959553bf3ede72.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:38:43.309651 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.296267 6511 generic.go:356] "Generic (PLEG): container finished" podID="6e865720-d98b-40e3-8bbc-3b63bd04f618" containerID="09817dafc1a1c2ece03761d67e3529b0125f50e686c5a94670959553bf3ede72" exitCode=0 Oct 30 19:38:43.309651 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.296343 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" event={"ID":"6e865720-d98b-40e3-8bbc-3b63bd04f618","Type":"ContainerDied","Data":"09817dafc1a1c2ece03761d67e3529b0125f50e686c5a94670959553bf3ede72"} Oct 30 19:38:43.422477 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.421937 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:38:43.451405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.448953 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs\") pod \"6e865720-d98b-40e3-8bbc-3b63bd04f618\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " Oct 30 19:38:43.451405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.449116 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls\") pod \"6e865720-d98b-40e3-8bbc-3b63bd04f618\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " Oct 30 19:38:43.451405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.449196 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca\") pod \"6e865720-d98b-40e3-8bbc-3b63bd04f618\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " Oct 30 19:38:43.451405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.449225 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd\" (UniqueName: \"kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd\") pod \"6e865720-d98b-40e3-8bbc-3b63bd04f618\" (UID: \"6e865720-d98b-40e3-8bbc-3b63bd04f618\") " Oct 30 19:38:43.451405 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.450548 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca" pod "6e865720-d98b-40e3-8bbc-3b63bd04f618" (UID: "6e865720-d98b-40e3-8bbc-3b63bd04f618"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:43.451886 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.451847 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs" pod "6e865720-d98b-40e3-8bbc-3b63bd04f618" (UID: "6e865720-d98b-40e3-8bbc-3b63bd04f618"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:43.459448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.457574 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd" pod "6e865720-d98b-40e3-8bbc-3b63bd04f618" (UID: "6e865720-d98b-40e3-8bbc-3b63bd04f618"). InnerVolumeSpecName "kube-api-access-ltqzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:43.459448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.457698 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls" pod "6e865720-d98b-40e3-8bbc-3b63bd04f618" (UID: "6e865720-d98b-40e3-8bbc-3b63bd04f618"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:43.550666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.550635 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:43.550666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.550666 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e865720-d98b-40e3-8bbc-3b63bd04f618-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:43.550881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.550681 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltqzd\" (UniqueName: \"kubernetes.io/projected/6e865720-d98b-40e3-8bbc-3b63bd04f618-kube-api-access-ltqzd\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:43.550881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:43.550697 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e865720-d98b-40e3-8bbc-3b63bd04f618-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:44.306506 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:44.306474 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" Oct 30 19:38:44.306506 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:44.306483 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r" event={"ID":"6e865720-d98b-40e3-8bbc-3b63bd04f618","Type":"ContainerDied","Data":"67cba3b2bab747f6c81b4b0f0da5719483ae21439293117afd902bc7e681322a"} Oct 30 19:38:44.307142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:44.306547 6511 scope.go:122] "RemoveContainer" containerID="09817dafc1a1c2ece03761d67e3529b0125f50e686c5a94670959553bf3ede72" Oct 30 19:38:44.376477 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:44.376383 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r"] Oct 30 19:38:44.392278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:44.392230 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6a6348-7845fc7c44-zmc4r"] Oct 30 19:38:46.101271 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:46.101235 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e865720-d98b-40e3-8bbc-3b63bd04f618" path="/var/lib/kubelet/pods/6e865720-d98b-40e3-8bbc-3b63bd04f618/volumes" Oct 30 19:38:48.665568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.665438 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867"] Oct 30 19:38:48.666250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.666231 6511 cpu_manager.go:414] "containerMap: removing container" podUID="6e865720-d98b-40e3-8bbc-3b63bd04f618" containerName="maas-api" Oct 30 19:38:48.666373 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.666253 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e865720-d98b-40e3-8bbc-3b63bd04f618" containerName="maas-api" Oct 30 19:38:48.666509 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.666494 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e865720-d98b-40e3-8bbc-3b63bd04f618" containerName="maas-api" Oct 30 19:38:48.685351 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.680231 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.685351 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.683101 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867"] Oct 30 19:38:48.706010 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.705973 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1e13df-serving-cert\"" Oct 30 19:38:48.746989 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.746956 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qpw\" (UniqueName: \"kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.747151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.747115 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.747206 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.747171 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.747260 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.747228 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.848630 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.848585 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qpw\" (UniqueName: \"kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.848852 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.848743 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.848852 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.848795 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.850828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.850792 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.852598 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.852567 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.855809 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.855751 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.856343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.856318 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:48.861475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.860998 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qpw\" (UniqueName: \"kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw\") pod \"maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:49.000005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:48.999908 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:49.291009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:49.290974 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867"] Oct 30 19:38:49.292042 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:38:49.291988 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4897e42d_9776_4111_9df9_73622a74f67c.slice/crio-f181b2d31bc8b61c0ae6e8445c3ac9ffe8643eb9f7ba57a8d6f45c9812c8edba WatchSource:0}: Error finding container f181b2d31bc8b61c0ae6e8445c3ac9ffe8643eb9f7ba57a8d6f45c9812c8edba: Status 404 returned error can't find the container with id f181b2d31bc8b61c0ae6e8445c3ac9ffe8643eb9f7ba57a8d6f45c9812c8edba Oct 30 19:38:49.343619 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:49.343588 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" event={"ID":"4897e42d-9776-4111-9df9-73622a74f67c","Type":"ContainerStarted","Data":"f181b2d31bc8b61c0ae6e8445c3ac9ffe8643eb9f7ba57a8d6f45c9812c8edba"} Oct 30 19:38:50.351899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:50.351861 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" event={"ID":"4897e42d-9776-4111-9df9-73622a74f67c","Type":"ContainerStarted","Data":"c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13"} Oct 30 19:38:50.352280 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:50.351944 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:50.373519 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:50.373468 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" podStartSLOduration=1.9423874639999998 podStartE2EDuration="2.373455386s" totalImagesPullingTime="431.067922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:50.37110065 +0000 UTC m=+1304.394704552" watchObservedRunningTime="2026-10-30 19:38:50.373455386 +0000 UTC m=+1304.397059285" Oct 30 19:38:54.382208 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:38:54.382164 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:38:56.385232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:56.385199 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:38:56.885038 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:56.885004 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg"] Oct 30 19:38:56.885822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:56.885786 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" podUID="31558791-85f3-4319-be66-e9c51095b1e3" containerName="maas-api" containerID="cri-o://378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8" gracePeriod=30 Oct 30 19:38:57.255439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.253560 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:38:57.339034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.338997 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs\") pod \"31558791-85f3-4319-be66-e9c51095b1e3\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " Oct 30 19:38:57.339397 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.339363 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\") pod \"31558791-85f3-4319-be66-e9c51095b1e3\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " Oct 30 19:38:57.340231 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.340205 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf\" (UniqueName: \"kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf\") pod \"31558791-85f3-4319-be66-e9c51095b1e3\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " Oct 30 19:38:57.340541 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.340519 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca\") pod \"31558791-85f3-4319-be66-e9c51095b1e3\" (UID: \"31558791-85f3-4319-be66-e9c51095b1e3\") " Oct 30 19:38:57.350634 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.350605 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca" pod "31558791-85f3-4319-be66-e9c51095b1e3" (UID: "31558791-85f3-4319-be66-e9c51095b1e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:57.372662 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.372621 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls" pod "31558791-85f3-4319-be66-e9c51095b1e3" (UID: "31558791-85f3-4319-be66-e9c51095b1e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:57.376623 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.376591 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs" pod "31558791-85f3-4319-be66-e9c51095b1e3" (UID: "31558791-85f3-4319-be66-e9c51095b1e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:57.382047 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.382015 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf" pod "31558791-85f3-4319-be66-e9c51095b1e3" (UID: "31558791-85f3-4319-be66-e9c51095b1e3"). InnerVolumeSpecName "kube-api-access-vhbtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:57.432192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.432151 6511 generic.go:356] "Generic (PLEG): container finished" podID="31558791-85f3-4319-be66-e9c51095b1e3" containerID="378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8" exitCode=0 Oct 30 19:38:57.432716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.432208 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" event={"ID":"31558791-85f3-4319-be66-e9c51095b1e3","Type":"ContainerDied","Data":"378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8"} Oct 30 19:38:57.432716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.432256 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" event={"ID":"31558791-85f3-4319-be66-e9c51095b1e3","Type":"ContainerDied","Data":"8b869c3404a480e783d1f9aaf8148b8e3cc62f397c27a90abe396549c417c9da"} Oct 30 19:38:57.432716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.432291 6511 scope.go:122] "RemoveContainer" containerID="378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8" Oct 30 19:38:57.432716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.432460 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg" Oct 30 19:38:57.443839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.443667 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31558791-85f3-4319-be66-e9c51095b1e3-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:57.443839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.443698 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:57.443839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.443714 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31558791-85f3-4319-be66-e9c51095b1e3-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:57.443839 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.443729 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbtf\" (UniqueName: \"kubernetes.io/projected/31558791-85f3-4319-be66-e9c51095b1e3-kube-api-access-vhbtf\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:57.447021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.447002 6511 scope.go:122] "RemoveContainer" containerID="378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8" Oct 30 19:38:57.447588 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:38:57.447525 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8\": container with ID starting with 378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8 not found: ID does not exist" containerID="378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8" Oct 30 19:38:57.447588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.447561 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8"} err="failed to get container status \"378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8\": rpc error: code = NotFound desc = could not find container \"378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8\": container with ID starting with 378bd2d9a7c574fd233147296727bf71bbdff83900a2156dcbcf098e85fdfea8 not found: ID does not exist" Oct 30 19:38:57.463957 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.463926 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg"] Oct 30 19:38:57.471032 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:57.471003 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-ae425f-5f84f8d48b-rlrxg"] Oct 30 19:38:58.098389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:38:58.098351 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31558791-85f3-4319-be66-e9c51095b1e3" path="/var/lib/kubelet/pods/31558791-85f3-4319-be66-e9c51095b1e3/volumes" Oct 30 19:39:05.392470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.392433 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg"] Oct 30 19:39:05.393326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.393300 6511 cpu_manager.go:414] "containerMap: removing container" podUID="31558791-85f3-4319-be66-e9c51095b1e3" containerName="maas-api" Oct 30 19:39:05.393326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.393326 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31558791-85f3-4319-be66-e9c51095b1e3" containerName="maas-api" Oct 30 19:39:05.393568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.393535 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="31558791-85f3-4319-be66-e9c51095b1e3" containerName="maas-api" Oct 30 19:39:05.398451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.398428 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.411380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.411338 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-22af66-serving-cert\"" Oct 30 19:39:05.414678 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.414655 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg"] Oct 30 19:39:05.465703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.465661 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.465877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.465729 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.465966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.465940 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.466048 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.466035 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4vr\" (UniqueName: \"kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.567242 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.567200 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.567461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.567309 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4vr\" (UniqueName: \"kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.567549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.567460 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.567549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.567533 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.568247 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.568220 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.570386 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.570363 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.570546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.570533 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.577860 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.577832 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4vr\" (UniqueName: \"kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr\") pod \"maas-api-e2e-api-b-22af66-6c748b7445-vc9sg\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:05.714028 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:05.713944 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:06.004243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:06.004116 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg"] Oct 30 19:39:06.004647 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:39:06.004619 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a662714_b2ec_421e_83ec_ca6ad917c8f8.slice/crio-59e3b09662f2c64f838a7178ae440b783f66382906a97994efb5b17cb7aa8e28 WatchSource:0}: Error finding container 59e3b09662f2c64f838a7178ae440b783f66382906a97994efb5b17cb7aa8e28: Status 404 returned error can't find the container with id 59e3b09662f2c64f838a7178ae440b783f66382906a97994efb5b17cb7aa8e28 Oct 30 19:39:06.499684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:06.499652 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" event={"ID":"6a662714-b2ec-421e-83ec-ca6ad917c8f8","Type":"ContainerStarted","Data":"59e3b09662f2c64f838a7178ae440b783f66382906a97994efb5b17cb7aa8e28"} Oct 30 19:39:07.509010 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:07.508974 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" event={"ID":"6a662714-b2ec-421e-83ec-ca6ad917c8f8","Type":"ContainerStarted","Data":"739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851"} Oct 30 19:39:07.509503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:07.509164 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:07.537792 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:07.537742 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" podStartSLOduration=2.117233542 podStartE2EDuration="2.537728222s" totalImagesPullingTime="420.49468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:07.536233798 +0000 UTC m=+1321.559837698" watchObservedRunningTime="2026-10-30 19:39:07.537728222 +0000 UTC m=+1321.561332121" Oct 30 19:39:08.310198 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:08.310160 6511 scope.go:122] "RemoveContainer" containerID="8c4577b0312e2373b29eebd628d3e5fe27dda56571196763470b59699e4e2d3a" Oct 30 19:39:08.322172 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:08.322154 6511 scope.go:122] "RemoveContainer" containerID="9723fc2ff6fa1e7b08971e9f7ffb7f442e3cc832b80f697b89b878edaa25465b" Oct 30 19:39:11.908430 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:11.908381 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26"] Oct 30 19:39:13.520535 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:13.520502 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:39:15.587576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.587538 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98"] Oct 30 19:39:15.594607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.594585 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.599773 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.599747 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-744b57-serving-cert\"" Oct 30 19:39:15.603785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.603760 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98"] Oct 30 19:39:15.706502 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.706470 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.706654 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.706515 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.706654 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.706624 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.706737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.706720 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfmk\" (UniqueName: \"kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.808050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.807816 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.808050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.807884 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.808050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.807943 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.808050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.807999 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfmk\" (UniqueName: \"kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.808899 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.808814 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.810968 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.810944 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.811193 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.811169 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.817633 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.817204 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfmk\" (UniqueName: \"kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk\") pod \"maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:15.915637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:15.915543 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:16.208514 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:39:16.208484 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606e46d4_427c_4d9b_b63f_efecaa8d0849.slice/crio-cb313bff7c41c3b7f2bddc8ed0533a57a338879fa0e242726bb7b7edcb8fff7f WatchSource:0}: Error finding container cb313bff7c41c3b7f2bddc8ed0533a57a338879fa0e242726bb7b7edcb8fff7f: Status 404 returned error can't find the container with id cb313bff7c41c3b7f2bddc8ed0533a57a338879fa0e242726bb7b7edcb8fff7f Oct 30 19:39:16.211088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:16.211021 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98"] Oct 30 19:39:16.575909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:16.575879 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" event={"ID":"606e46d4-427c-4d9b-b63f-efecaa8d0849","Type":"ContainerStarted","Data":"cb313bff7c41c3b7f2bddc8ed0533a57a338879fa0e242726bb7b7edcb8fff7f"} Oct 30 19:39:17.592436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:17.592327 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" event={"ID":"606e46d4-427c-4d9b-b63f-efecaa8d0849","Type":"ContainerStarted","Data":"d5d3c91fa965db11e88be1496989f4bc9a8387bf0bccadba910957287f5ee857"} Oct 30 19:39:17.597392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:17.593401 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:17.642434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:17.641626 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" podStartSLOduration=2.1609434690000002 podStartE2EDuration="2.641606792s" totalImagesPullingTime="480.663323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:17.638944669 +0000 UTC m=+1331.662548569" watchObservedRunningTime="2026-10-30 19:39:17.641606792 +0000 UTC m=+1331.665210691" Oct 30 19:39:24.384954 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:39:24.384908 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:39:24.614207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:24.614172 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:39:26.931873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:26.931835 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" containerID="cri-o://e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0" gracePeriod=45 Oct 30 19:39:27.216778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.216755 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:39:27.293574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293538 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm\") pod \"df8574bd-d1a6-4b82-8004-5a719d825265\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " Oct 30 19:39:27.293574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293573 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir\") pod \"df8574bd-d1a6-4b82-8004-5a719d825265\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " Oct 30 19:39:27.293823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293685 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home\") pod \"df8574bd-d1a6-4b82-8004-5a719d825265\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " Oct 30 19:39:27.293823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293759 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs\" (UniqueName: \"kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs\") pod \"df8574bd-d1a6-4b82-8004-5a719d825265\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " Oct 30 19:39:27.293969 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293947 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir" pod "df8574bd-d1a6-4b82-8004-5a719d825265" (UID: "df8574bd-d1a6-4b82-8004-5a719d825265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:27.294034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293971 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home" pod "df8574bd-d1a6-4b82-8004-5a719d825265" (UID: "df8574bd-d1a6-4b82-8004-5a719d825265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:27.294034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.293992 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache\") pod \"df8574bd-d1a6-4b82-8004-5a719d825265\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " Oct 30 19:39:27.294034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.294018 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c\" (UniqueName: \"kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c\") pod \"df8574bd-d1a6-4b82-8004-5a719d825265\" (UID: \"df8574bd-d1a6-4b82-8004-5a719d825265\") " Oct 30 19:39:27.294228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.294208 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache" pod "df8574bd-d1a6-4b82-8004-5a719d825265" (UID: "df8574bd-d1a6-4b82-8004-5a719d825265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:27.294616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.294586 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:27.294616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.294614 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:27.294778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.294629 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:27.296321 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.296301 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs" pod "df8574bd-d1a6-4b82-8004-5a719d825265" (UID: "df8574bd-d1a6-4b82-8004-5a719d825265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:27.296450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.296323 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm" pod "df8574bd-d1a6-4b82-8004-5a719d825265" (UID: "df8574bd-d1a6-4b82-8004-5a719d825265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:39:27.296450 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.296346 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c" pod "df8574bd-d1a6-4b82-8004-5a719d825265" (UID: "df8574bd-d1a6-4b82-8004-5a719d825265"). InnerVolumeSpecName "kube-api-access-lfr6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:27.396539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.396507 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df8574bd-d1a6-4b82-8004-5a719d825265-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:27.396745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.396583 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfr6c\" (UniqueName: \"kubernetes.io/projected/df8574bd-d1a6-4b82-8004-5a719d825265-kube-api-access-lfr6c\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:27.396745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.396603 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df8574bd-d1a6-4b82-8004-5a719d825265-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:27.673764 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.673728 6511 generic.go:356] "Generic (PLEG): container finished" podID="df8574bd-d1a6-4b82-8004-5a719d825265" containerID="e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0" exitCode=0 Oct 30 19:39:27.673764 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.673768 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" Oct 30 19:39:27.673972 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.673775 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" event={"ID":"df8574bd-d1a6-4b82-8004-5a719d825265","Type":"ContainerDied","Data":"e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0"} Oct 30 19:39:27.673972 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.673812 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" event={"ID":"df8574bd-d1a6-4b82-8004-5a719d825265","Type":"ContainerDied","Data":"904feda20aa432dd65be51af08c4c5c643168ef72f1528a2a0549998816cf707"} Oct 30 19:39:27.673972 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.673836 6511 scope.go:122] "RemoveContainer" containerID="e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0" Oct 30 19:39:27.685200 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.685180 6511 scope.go:122] "RemoveContainer" containerID="e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0" Oct 30 19:39:27.685496 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:39:27.685474 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0\": container with ID starting with e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0 not found: ID does not exist" containerID="e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0" Oct 30 19:39:27.685586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.685503 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0"} err="failed to get container status \"e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0\": rpc error: code = NotFound desc = could not find container \"e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0\": container with ID starting with e939da4c803d83e62324747ad23c1013d5a8818f6d4ba5f8a3e6c0947db75fb0 not found: ID does not exist" Oct 30 19:39:27.710647 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.710615 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26"] Oct 30 19:39:27.739249 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:27.739220 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26"] Oct 30 19:39:28.097867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:28.097836 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" path="/var/lib/kubelet/pods/df8574bd-d1a6-4b82-8004-5a719d825265/volumes" Oct 30 19:39:28.118180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:28.118145 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ae63bf/distinct-2-w0-ae63bf-kserve-788c8f7d67-d9g26" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:45.127085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.127045 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk"] Oct 30 19:39:45.127664 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.127303 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" podUID="715bf9e2-bae1-4b7f-a55e-b789286ad80b" containerName="maas-api" containerID="cri-o://dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561" gracePeriod=30 Oct 30 19:39:45.590504 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.590461 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:39:45.710460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.708921 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs\" (UniqueName: \"kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs\") pod \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " Oct 30 19:39:45.710460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.708967 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs\") pod \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " Oct 30 19:39:45.710460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.709024 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls\") pod \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " Oct 30 19:39:45.710460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.709226 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca\") pod \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\" (UID: \"715bf9e2-bae1-4b7f-a55e-b789286ad80b\") " Oct 30 19:39:45.710460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.710312 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca" pod "715bf9e2-bae1-4b7f-a55e-b789286ad80b" (UID: "715bf9e2-bae1-4b7f-a55e-b789286ad80b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:45.712138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.712072 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs" pod "715bf9e2-bae1-4b7f-a55e-b789286ad80b" (UID: "715bf9e2-bae1-4b7f-a55e-b789286ad80b"). InnerVolumeSpecName "kube-api-access-8vkgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:45.712576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.712550 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls" pod "715bf9e2-bae1-4b7f-a55e-b789286ad80b" (UID: "715bf9e2-bae1-4b7f-a55e-b789286ad80b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:45.713460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.713431 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs" pod "715bf9e2-bae1-4b7f-a55e-b789286ad80b" (UID: "715bf9e2-bae1-4b7f-a55e-b789286ad80b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:45.811201 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.811169 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715bf9e2-bae1-4b7f-a55e-b789286ad80b-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:45.811201 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.811198 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vkgs\" (UniqueName: \"kubernetes.io/projected/715bf9e2-bae1-4b7f-a55e-b789286ad80b-kube-api-access-8vkgs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:45.811392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.811207 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:45.811392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.811216 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715bf9e2-bae1-4b7f-a55e-b789286ad80b-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:45.828070 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.828033 6511 generic.go:356] "Generic (PLEG): container finished" podID="715bf9e2-bae1-4b7f-a55e-b789286ad80b" containerID="dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561" exitCode=0 Oct 30 19:39:45.828236 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.828094 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" Oct 30 19:39:45.828236 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.828113 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" event={"ID":"715bf9e2-bae1-4b7f-a55e-b789286ad80b","Type":"ContainerDied","Data":"dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561"} Oct 30 19:39:45.828236 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.828174 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk" event={"ID":"715bf9e2-bae1-4b7f-a55e-b789286ad80b","Type":"ContainerDied","Data":"d3d9978ade83a0173681bd2d6bb0066a24c88174f9123bafd725693ce0754670"} Oct 30 19:39:45.828236 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.828206 6511 scope.go:122] "RemoveContainer" containerID="dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561" Oct 30 19:39:45.838879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.838862 6511 scope.go:122] "RemoveContainer" containerID="dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561" Oct 30 19:39:45.839138 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:39:45.839119 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561\": container with ID starting with dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561 not found: ID does not exist" containerID="dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561" Oct 30 19:39:45.839210 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.839149 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561"} err="failed to get container status \"dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561\": rpc error: code = NotFound desc = could not find container \"dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561\": container with ID starting with dc46aed22d907ad4e9030ce4041de72d1601bb02f410ffeaf7cb5cdb7cb19561 not found: ID does not exist" Oct 30 19:39:45.884086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.882292 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk"] Oct 30 19:39:45.886395 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:45.886359 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ae63bf-594d7688fb-qzpxk"] Oct 30 19:39:46.097891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:46.097812 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715bf9e2-bae1-4b7f-a55e-b789286ad80b" path="/var/lib/kubelet/pods/715bf9e2-bae1-4b7f-a55e-b789286ad80b/volumes" Oct 30 19:39:54.390518 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:39:54.390478 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:39:55.508134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:55.508094 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6"] Oct 30 19:39:55.509053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:55.508967 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" containerID="cri-o://3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b" gracePeriod=30 Oct 30 19:39:55.514241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:55.514215 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb"] Oct 30 19:39:55.514536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:55.514498 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" containerID="cri-o://356d2a1cfec1a88bed27d05f6ca37057b400273508677404dc99686b1a54483c" gracePeriod=30 Oct 30 19:39:56.389745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:56.389710 6511 patch_prober.go:29] interesting pod/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:39:56.389907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:56.389770 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:39:58.517155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:58.517114 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh"] Oct 30 19:39:58.517741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:39:58.517393 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" containerID="cri-o://60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2" gracePeriod=30 Oct 30 19:40:00.861451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:00.861129 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:40:01.011313 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.011275 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.011521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.011327 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.011521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.011397 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.011658 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.011556 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.011851 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.011823 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.012000 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.011979 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.012074 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012019 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.012074 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012063 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.012182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012127 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.012182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012150 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.012285 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012193 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy\") pod \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\" (UID: \"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652\") " Oct 30 19:40:01.013433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012781 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.013433 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.012809 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.017835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.017803 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:01.018114 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.018049 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.018202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.018134 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.034261 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.034658 6511 generic.go:356] "Generic (PLEG): container finished" podID="571df9c4-0659-4745-914b-8f36babc24e8" containerID="356d2a1cfec1a88bed27d05f6ca37057b400273508677404dc99686b1a54483c" exitCode=0 Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.034843 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" event={"ID":"571df9c4-0659-4745-914b-8f36babc24e8","Type":"ContainerDied","Data":"356d2a1cfec1a88bed27d05f6ca37057b400273508677404dc99686b1a54483c"} Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.034906 6511 scope.go:122] "RemoveContainer" containerID="c165bcc42a53926f204f025fc5175da4384ec82713fd864d38aba3fd89fadd14" Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.035569 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.037238 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.037570 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4" pod "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" (UID: "46e1e1a6-2e94-4f02-8e5f-0b48d2abb652"). InnerVolumeSpecName "kube-api-access-67kj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.039763 6511 generic.go:356] "Generic (PLEG): container finished" podID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerID="3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b" exitCode=0 Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.039870 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" event={"ID":"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652","Type":"ContainerDied","Data":"3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b"} Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.039918 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" event={"ID":"46e1e1a6-2e94-4f02-8e5f-0b48d2abb652","Type":"ContainerDied","Data":"36779852c9b048a59c67b00646f01ec1e8c36bf627f2569c23e5e606626986ba"} Oct 30 19:40:01.040451 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.040003 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6" Oct 30 19:40:01.059222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.059199 6511 scope.go:122] "RemoveContainer" containerID="3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b" Oct 30 19:40:01.061079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.061056 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:40:01.069992 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.069974 6511 scope.go:122] "RemoveContainer" containerID="3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b" Oct 30 19:40:01.070269 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:40:01.070249 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b\": container with ID starting with 3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b not found: ID does not exist" containerID="3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b" Oct 30 19:40:01.070344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.070278 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b"} err="failed to get container status \"3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b\": rpc error: code = NotFound desc = could not find container \"3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b\": container with ID starting with 3432c69200a0c1219e16fc63de7cddc6dcf8ad2176663c0b058cdd6790c4084b not found: ID does not exist" Oct 30 19:40:01.078678 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.078653 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6"] Oct 30 19:40:01.083781 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.083741 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-6smj6"] Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114098 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114207 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114240 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114316 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114398 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114476 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114505 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114585 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114673 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token\") pod \"571df9c4-0659-4745-914b-8f36babc24e8\" (UID: \"571df9c4-0659-4745-914b-8f36babc24e8\") " Oct 30 19:40:01.114829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114790 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114970 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.114994 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115145 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115187 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115364 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115383 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115396 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115442 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115458 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115472 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115489 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115502 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115516 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kj4\" (UniqueName: \"kubernetes.io/projected/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-kube-api-access-67kj4\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115531 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115545 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.115579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.115560 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571df9c4-0659-4745-914b-8f36babc24e8-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.118188 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.118160 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.118388 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.118370 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "kube-api-access-vsrjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:01.118670 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.118589 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:01.120105 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.120075 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo" pod "571df9c4-0659-4745-914b-8f36babc24e8" (UID: "571df9c4-0659-4745-914b-8f36babc24e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:01.217043 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.217009 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrjj\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-kube-api-access-vsrjj\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.217043 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.217042 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571df9c4-0659-4745-914b-8f36babc24e8-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.217227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.217054 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571df9c4-0659-4745-914b-8f36babc24e8-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:01.217227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:01.217065 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571df9c4-0659-4745-914b-8f36babc24e8-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:02.047515 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:02.047480 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" event={"ID":"571df9c4-0659-4745-914b-8f36babc24e8","Type":"ContainerDied","Data":"457bf28bd30820f38c06f022f9cd95777dd73a05fc273e72ac51041b1cef13af"} Oct 30 19:40:02.047971 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:02.047481 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb" Oct 30 19:40:02.047971 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:02.047572 6511 scope.go:122] "RemoveContainer" containerID="356d2a1cfec1a88bed27d05f6ca37057b400273508677404dc99686b1a54483c" Oct 30 19:40:02.098246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:02.098217 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" path="/var/lib/kubelet/pods/46e1e1a6-2e94-4f02-8e5f-0b48d2abb652/volumes" Oct 30 19:40:02.098540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:02.098526 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb"] Oct 30 19:40:02.099703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:02.099683 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ae63bf-openshift-default-556d46cdff-8pztb"] Oct 30 19:40:03.815635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.815606 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:40:03.952085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952052 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952100 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952143 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952191 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952230 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952280 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952327 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952342 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:03.952606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952373 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952551 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t\") pod \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\" (UID: \"32977db8-7d4d-4fdd-a156-f87fb7cdd079\") " Oct 30 19:40:03.952606 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.952578 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:03.953304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.953145 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:03.953695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.953676 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:03.953846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.953190 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:03.953965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.953756 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:03.954130 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.954100 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:03.956014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.955988 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:03.956505 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.956477 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "kube-api-access-jxz7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:03.956784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.956677 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:03.956784 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:03.956731 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token" pod "32977db8-7d4d-4fdd-a156-f87fb7cdd079" (UID: "32977db8-7d4d-4fdd-a156-f87fb7cdd079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:04.055171 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055137 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxz7t\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-kube-api-access-jxz7t\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.055331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055177 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.055331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055194 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.055331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055210 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.055331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055225 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32977db8-7d4d-4fdd-a156-f87fb7cdd079-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.055331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055239 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.055331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.055256 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32977db8-7d4d-4fdd-a156-f87fb7cdd079-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:04.063865 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.063838 6511 generic.go:356] "Generic (PLEG): container finished" podID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerID="60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2" exitCode=0 Oct 30 19:40:04.063991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.063897 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" Oct 30 19:40:04.063991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.063922 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" event={"ID":"32977db8-7d4d-4fdd-a156-f87fb7cdd079","Type":"ContainerDied","Data":"60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2"} Oct 30 19:40:04.064064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.063999 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh" event={"ID":"32977db8-7d4d-4fdd-a156-f87fb7cdd079","Type":"ContainerDied","Data":"bce4d70a6940cc8788efbd249c693df1474e269dcd3ad347a98437bb1bb88009"} Oct 30 19:40:04.064064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.064023 6511 scope.go:122] "RemoveContainer" containerID="60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2" Oct 30 19:40:04.075388 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.075371 6511 scope.go:122] "RemoveContainer" containerID="6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca" Oct 30 19:40:04.086055 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.086037 6511 scope.go:122] "RemoveContainer" containerID="60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2" Oct 30 19:40:04.086534 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:40:04.086390 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2\": container with ID starting with 60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2 not found: ID does not exist" containerID="60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2" Oct 30 19:40:04.086534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.086476 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2"} err="failed to get container status \"60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2\": rpc error: code = NotFound desc = could not find container \"60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2\": container with ID starting with 60e975ec243d575a6963ea292e9506ba8c96552d188a7e583cd4b17daf9af9e2 not found: ID does not exist" Oct 30 19:40:04.086534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.086526 6511 scope.go:122] "RemoveContainer" containerID="6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca" Oct 30 19:40:04.088011 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:40:04.087529 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca\": container with ID starting with 6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca not found: ID does not exist" containerID="6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca" Oct 30 19:40:04.088011 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.087567 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca"} err="failed to get container status \"6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca\": rpc error: code = NotFound desc = could not find container \"6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca\": container with ID starting with 6c7ee6670355cd890500cc00f559ae742619137dd5248bcfcb6da03d694a8bca not found: ID does not exist" Oct 30 19:40:04.103973 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.103944 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571df9c4-0659-4745-914b-8f36babc24e8" path="/var/lib/kubelet/pods/571df9c4-0659-4745-914b-8f36babc24e8/volumes" Oct 30 19:40:04.104256 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.104243 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh"] Oct 30 19:40:04.104305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:04.104275 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-5bwkh"] Oct 30 19:40:06.097718 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:06.097682 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" path="/var/lib/kubelet/pods/32977db8-7d4d-4fdd-a156-f87fb7cdd079/volumes" Oct 30 19:40:13.478279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.478238 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9"] Oct 30 19:40:13.478733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.478497 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" podUID="f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" containerName="maas-api" containerID="cri-o://4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b" gracePeriod=30 Oct 30 19:40:13.756962 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.756935 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:40:13.898089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.898056 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls\") pod \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " Oct 30 19:40:13.898628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.898603 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw\" (UniqueName: \"kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw\") pod \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " Oct 30 19:40:13.898811 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.898796 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs\") pod \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " Oct 30 19:40:13.898969 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.898953 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca\") pod \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\" (UID: \"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8\") " Oct 30 19:40:13.899896 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.899871 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca" pod "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" (UID: "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:13.901510 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.901488 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls" pod "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" (UID: "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:13.901774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.901754 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs" pod "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" (UID: "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:13.902820 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:13.902800 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw" pod "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" (UID: "f754f0de-30f6-4ef5-b1d9-67ab5d827fd8"). InnerVolumeSpecName "kube-api-access-h5xpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:14.001084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.001002 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xpw\" (UniqueName: \"kubernetes.io/projected/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-kube-api-access-h5xpw\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:14.001084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.001033 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:14.001084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.001043 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:14.001084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.001054 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:14.135610 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.135574 6511 generic.go:356] "Generic (PLEG): container finished" podID="f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" containerID="4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b" exitCode=0 Oct 30 19:40:14.135798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.135620 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" Oct 30 19:40:14.135798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.135654 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" event={"ID":"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8","Type":"ContainerDied","Data":"4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b"} Oct 30 19:40:14.135798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.135711 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9" event={"ID":"f754f0de-30f6-4ef5-b1d9-67ab5d827fd8","Type":"ContainerDied","Data":"8447da6c2a37b8d40e21414c8857e4d30cdb8b1aef679d08188263b37e70b462"} Oct 30 19:40:14.135798 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.135732 6511 scope.go:122] "RemoveContainer" containerID="4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b" Oct 30 19:40:14.146404 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.146380 6511 scope.go:122] "RemoveContainer" containerID="4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b" Oct 30 19:40:14.146745 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:40:14.146718 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b\": container with ID starting with 4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b not found: ID does not exist" containerID="4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b" Oct 30 19:40:14.146844 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.146753 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b"} err="failed to get container status \"4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b\": rpc error: code = NotFound desc = could not find container \"4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b\": container with ID starting with 4106a600d2bdf03cbeb8734a87365d5a808f453f12860483c58139bcf4c79e6b not found: ID does not exist" Oct 30 19:40:14.166393 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.166367 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9"] Oct 30 19:40:14.171400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:14.171379 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d1d256-665d4bbb95-k74z9"] Oct 30 19:40:16.115901 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:16.115859 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" path="/var/lib/kubelet/pods/f754f0de-30f6-4ef5-b1d9-67ab5d827fd8/volumes" Oct 30 19:40:23.247102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247064 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn"] Oct 30 19:40:23.247812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247780 6511 cpu_manager.go:414] "containerMap: removing container" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" Oct 30 19:40:23.247812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247804 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247822 6511 cpu_manager.go:414] "containerMap: removing container" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247829 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247841 6511 cpu_manager.go:414] "containerMap: removing container" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247847 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247865 6511 cpu_manager.go:414] "containerMap: removing container" podUID="715bf9e2-bae1-4b7f-a55e-b789286ad80b" containerName="maas-api" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247872 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715bf9e2-bae1-4b7f-a55e-b789286ad80b" containerName="maas-api" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247885 6511 cpu_manager.go:414] "containerMap: removing container" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247891 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247903 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" containerName="maas-api" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247909 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" containerName="maas-api" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247927 6511 cpu_manager.go:414] "containerMap: removing container" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247934 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247951 6511 cpu_manager.go:414] "containerMap: removing container" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" Oct 30 19:40:23.248014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.247958 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" Oct 30 19:40:23.248693 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.248101 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" Oct 30 19:40:23.248693 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.248116 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e1e1a6-2e94-4f02-8e5f-0b48d2abb652" containerName="istio-proxy" Oct 30 19:40:23.248693 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.248126 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8574bd-d1a6-4b82-8004-5a719d825265" containerName="main" Oct 30 19:40:23.248693 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.248138 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="715bf9e2-bae1-4b7f-a55e-b789286ad80b" containerName="maas-api" Oct 30 19:40:23.248693 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.248149 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" Oct 30 19:40:23.248693 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.248157 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="f754f0de-30f6-4ef5-b1d9-67ab5d827fd8" containerName="maas-api" Oct 30 19:40:23.253894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.253875 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.262522 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.262503 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3e35df-openshift-default-dockercfg-5snvf\"" Oct 30 19:40:23.265255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.265223 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn"] Oct 30 19:40:23.338310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338275 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338322 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338344 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338605 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338716 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338669 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8w5q\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338737 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338806 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338901 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338886 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.338958 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.338941 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.445267 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.445218 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.446030 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.446002 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.446294 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.446276 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.446460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.446444 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.446597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.446581 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.446786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.446768 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.446916 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.446899 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8w5q\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.447019 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.447005 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.447129 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.447114 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.447508 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.445904 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.447687 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.447668 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.494352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.447848 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.494631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.448056 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.494774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.448979 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.494872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.458960 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.494966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.476323 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.502014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.495230 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8w5q\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.502014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.501829 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token\") pod \"e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.573872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.573835 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:23.889162 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:40:23.889119 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0ffd8f_5549_49ed_9d8c_b30cc8474977.slice/crio-a3632718965c480950579e5c0ad8aa87aa260709e727199dc6060c8bea9ec9d9 WatchSource:0}: Error finding container a3632718965c480950579e5c0ad8aa87aa260709e727199dc6060c8bea9ec9d9: Status 404 returned error can't find the container with id a3632718965c480950579e5c0ad8aa87aa260709e727199dc6060c8bea9ec9d9 Oct 30 19:40:23.889862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.889832 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn"] Oct 30 19:40:23.892305 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.892259 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:40:23.892427 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.892352 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:40:23.892427 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:23.892399 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:40:24.237766 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:24.237677 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" event={"ID":"db0ffd8f-5549-49ed-9d8c-b30cc8474977","Type":"ContainerStarted","Data":"e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f"} Oct 30 19:40:24.237766 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:24.237759 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" event={"ID":"db0ffd8f-5549-49ed-9d8c-b30cc8474977","Type":"ContainerStarted","Data":"a3632718965c480950579e5c0ad8aa87aa260709e727199dc6060c8bea9ec9d9"} Oct 30 19:40:24.264146 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:24.264089 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podStartSLOduration=1.264070081 podStartE2EDuration="1.264070081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:24.262086997 +0000 UTC m=+1398.285690897" watchObservedRunningTime="2026-10-30 19:40:24.264070081 +0000 UTC m=+1398.287673983" Oct 30 19:40:24.395713 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:40:24.395662 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:40:24.574831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:24.574802 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:24.580838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:24.580817 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:25.245180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:25.245145 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:25.246185 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:25.246164 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:40:40.125605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.125558 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869"] Oct 30 19:40:40.126229 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.126212 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="571df9c4-0659-4745-914b-8f36babc24e8" containerName="istio-proxy" Oct 30 19:40:40.126229 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.126231 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="32977db8-7d4d-4fdd-a156-f87fb7cdd079" containerName="istio-proxy" Oct 30 19:40:40.129692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.129674 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.150060 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.150034 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3e35df-serving-cert\"" Oct 30 19:40:40.152723 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.152698 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869"] Oct 30 19:40:40.191983 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.191955 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.192132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.192030 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.192195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.192179 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vfw\" (UniqueName: \"kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.192254 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.192241 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.316206 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.316169 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vfw\" (UniqueName: \"kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.320542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.320508 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.323296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.323265 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.323536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.323501 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.324467 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.324169 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.340281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.340084 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.342621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.340672 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.342621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.341127 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vfw\" (UniqueName: \"kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw\") pod \"maas-api-e2e-inf-a-3e35df-85b957b874-gr869\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.444261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.444179 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:40.767391 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:40.760729 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869"] Oct 30 19:40:41.385096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:41.385063 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" event={"ID":"47ec0193-2f94-4575-b13f-a690ab6dc06e","Type":"ContainerStarted","Data":"3fc39258bc515e1effe164fbcce7b7ec9230222f2ddcb0b53f7c91d45be06aaf"} Oct 30 19:40:41.385522 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:41.385124 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" event={"ID":"47ec0193-2f94-4575-b13f-a690ab6dc06e","Type":"ContainerStarted","Data":"062e719aa1940435f6af090f841ac1ed3e77d8ec38e9d5e1028cdd1a0713a1ed"} Oct 30 19:40:41.385522 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:41.385152 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:41.406854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:41.406807 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" podStartSLOduration=0.959256872 podStartE2EDuration="1.406793997s" totalImagesPullingTime="447.537125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:41.404319737 +0000 UTC m=+1415.427923637" watchObservedRunningTime="2026-10-30 19:40:41.406793997 +0000 UTC m=+1415.430397937" Oct 30 19:40:42.119500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.119471 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq"] Oct 30 19:40:42.123493 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.123475 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.133178 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.133155 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1e13df-openshift-default-dockercfg-tkpw2\"" Oct 30 19:40:42.141995 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.141969 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq"] Oct 30 19:40:42.262109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262080 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9hw\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262126 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262196 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262346 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262265 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262346 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262297 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262434 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262361 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262471 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262536 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262524 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.262741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.262722 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363408 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363380 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363480 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363505 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9hw\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363539 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363566 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363593 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363619 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363863 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363738 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363863 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363792 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363852 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363892 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.363978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.363895 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.364139 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.364086 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.364364 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.364341 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.366021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.365998 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.366317 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.366298 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.372241 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.372186 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9hw\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.372382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.372363 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token\") pod \"e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.438062 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.438019 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:42.721573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.721541 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq"] Oct 30 19:40:42.722044 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:40:42.722010 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7864046_3736_42b7_9caa_2f0214187790.slice/crio-5ea6b95388be69a5b9ccb00cc751b39b40c13a3e7a8d3499beaccc6bb42dbe91 WatchSource:0}: Error finding container 5ea6b95388be69a5b9ccb00cc751b39b40c13a3e7a8d3499beaccc6bb42dbe91: Status 404 returned error can't find the container with id 5ea6b95388be69a5b9ccb00cc751b39b40c13a3e7a8d3499beaccc6bb42dbe91 Oct 30 19:40:42.724789 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.724754 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:40:42.724872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.724826 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:40:42.724872 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:42.724858 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:40:43.399215 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:43.399180 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" event={"ID":"c7864046-3736-42b7-9caa-2f0214187790","Type":"ContainerStarted","Data":"7d53d204c90a4201870c78121c3df5969eb929a733d45f43633fceab05c9ee32"} Oct 30 19:40:43.399402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:43.399240 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" event={"ID":"c7864046-3736-42b7-9caa-2f0214187790","Type":"ContainerStarted","Data":"5ea6b95388be69a5b9ccb00cc751b39b40c13a3e7a8d3499beaccc6bb42dbe91"} Oct 30 19:40:43.443607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:43.440007 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:43.444341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:43.444313 6511 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:40:43.444570 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:43.444542 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:40:43.447359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:43.447313 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podStartSLOduration=1.447300162 podStartE2EDuration="1.447300162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:43.440306005 +0000 UTC m=+1417.463909905" watchObservedRunningTime="2026-10-30 19:40:43.447300162 +0000 UTC m=+1417.470904060" Oct 30 19:40:44.444479 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:44.439832 6511 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:40:44.444479 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:44.439893 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:40:45.441441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:45.440786 6511 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:40:45.441441 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:45.440848 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:40:46.444438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:46.442582 6511 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:40:46.444438 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:46.442643 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:40:47.397701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:47.397673 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:40:47.453463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:47.443797 6511 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:40:47.453463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:47.443859 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:40:48.438838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:48.438796 6511 patch_prober.go:29] interesting pod/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:40:48.439138 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:48.439106 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:40:49.444147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:49.444113 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:49.444574 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:49.444543 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:49.445372 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:40:49.445351 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:40:54.385324 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:40:54.385265 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:41:05.623164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:05.623127 6511 generic.go:356] "Generic (PLEG): container finished" podID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerID="e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f" exitCode=0 Oct 30 19:41:05.623744 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:05.623208 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" event={"ID":"db0ffd8f-5549-49ed-9d8c-b30cc8474977","Type":"ContainerDied","Data":"e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f"} Oct 30 19:41:05.623824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:05.623799 6511 scope.go:122] "RemoveContainer" containerID="e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f" Oct 30 19:41:05.625646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:05.625616 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:41:05.625713 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:05.625692 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:41:05.625750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:05.625731 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:41:06.650160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:06.650127 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" event={"ID":"db0ffd8f-5549-49ed-9d8c-b30cc8474977","Type":"ContainerStarted","Data":"e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c"} Oct 30 19:41:07.574469 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:07.574400 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:41:07.575446 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:07.575403 6511 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:41:07.575573 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:07.575471 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 19:41:08.543437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.543164 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml"] Oct 30 19:41:08.549443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.549399 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.557814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.554675 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ac37c8-openshift-default-dockercfg-f7pmb\"" Oct 30 19:41:08.558513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.558489 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml"] Oct 30 19:41:08.574539 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.574505 6511 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:41:08.574680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.574581 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 19:41:08.705873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.705785 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.705873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.705825 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.705945 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.705988 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfh5\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.706010 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.706153 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706202 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.706189 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.706289 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.706350 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.706315 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.807736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.807696 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.807950 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.807766 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfh5\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.807950 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.807886 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.807966 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.807994 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808064 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808095 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808141 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808177 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808186 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808301 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808600 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808575 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808661 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808636 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.808889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.808868 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.810471 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.810454 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.810535 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.810505 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.840043 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.840009 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.861439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.855578 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfh5\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5\") pod \"e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:08.863245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.863208 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg"] Oct 30 19:41:08.863546 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.863490 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" podUID="6a662714-b2ec-421e-83ec-ca6ad917c8f8" containerName="maas-api" containerID="cri-o://739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851" gracePeriod=30 Oct 30 19:41:08.865879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:08.865835 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:09.456979 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.456927 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml"] Oct 30 19:41:09.472451 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:41:09.472424 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d7b3fb_886d_47fa_be1f_b6257175da60.slice/crio-9afee64380276442d9209b708a9377c56d74f78becde815198b3eb19e0d4d784 WatchSource:0}: Error finding container 9afee64380276442d9209b708a9377c56d74f78becde815198b3eb19e0d4d784: Status 404 returned error can't find the container with id 9afee64380276442d9209b708a9377c56d74f78becde815198b3eb19e0d4d784 Oct 30 19:41:09.475188 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.475151 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:41:09.475342 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.475327 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:41:09.475454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.475442 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:41:09.553497 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.553469 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:41:09.574523 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.574503 6511 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:41:09.574638 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.574545 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 19:41:09.682096 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.682009 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" event={"ID":"45d7b3fb-886d-47fa-be1f-b6257175da60","Type":"ContainerStarted","Data":"f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4"} Oct 30 19:41:09.682245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.682130 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" event={"ID":"45d7b3fb-886d-47fa-be1f-b6257175da60","Type":"ContainerStarted","Data":"9afee64380276442d9209b708a9377c56d74f78becde815198b3eb19e0d4d784"} Oct 30 19:41:09.684157 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.684128 6511 generic.go:356] "Generic (PLEG): container finished" podID="6a662714-b2ec-421e-83ec-ca6ad917c8f8" containerID="739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851" exitCode=0 Oct 30 19:41:09.684277 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.684167 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" Oct 30 19:41:09.684277 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.684242 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" event={"ID":"6a662714-b2ec-421e-83ec-ca6ad917c8f8","Type":"ContainerDied","Data":"739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851"} Oct 30 19:41:09.684361 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.684277 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg" event={"ID":"6a662714-b2ec-421e-83ec-ca6ad917c8f8","Type":"ContainerDied","Data":"59e3b09662f2c64f838a7178ae440b783f66382906a97994efb5b17cb7aa8e28"} Oct 30 19:41:09.684361 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.684304 6511 scope.go:122] "RemoveContainer" containerID="739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851" Oct 30 19:41:09.694818 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.694801 6511 scope.go:122] "RemoveContainer" containerID="739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851" Oct 30 19:41:09.695086 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:41:09.695063 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851\": container with ID starting with 739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851 not found: ID does not exist" containerID="739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851" Oct 30 19:41:09.695147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.695112 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851"} err="failed to get container status \"739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851\": rpc error: code = NotFound desc = could not find container \"739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851\": container with ID starting with 739a4c227f5f36f6b18027d9d6a7e91da019aab356eecdb42e5fbb77b491d851 not found: ID does not exist" Oct 30 19:41:09.698741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.698720 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca\") pod \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " Oct 30 19:41:09.698813 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.698783 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls\") pod \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " Oct 30 19:41:09.698902 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.698886 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs\") pod \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " Oct 30 19:41:09.698986 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.698970 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr\" (UniqueName: \"kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr\") pod \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\" (UID: \"6a662714-b2ec-421e-83ec-ca6ad917c8f8\") " Oct 30 19:41:09.699816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.699738 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca" pod "6a662714-b2ec-421e-83ec-ca6ad917c8f8" (UID: "6a662714-b2ec-421e-83ec-ca6ad917c8f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:09.701753 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.701719 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls" pod "6a662714-b2ec-421e-83ec-ca6ad917c8f8" (UID: "6a662714-b2ec-421e-83ec-ca6ad917c8f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:09.702104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.702085 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs" pod "6a662714-b2ec-421e-83ec-ca6ad917c8f8" (UID: "6a662714-b2ec-421e-83ec-ca6ad917c8f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:41:09.703581 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.703544 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr" pod "6a662714-b2ec-421e-83ec-ca6ad917c8f8" (UID: "6a662714-b2ec-421e-83ec-ca6ad917c8f8"). InnerVolumeSpecName "kube-api-access-rt4vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:09.713244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.713204 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podStartSLOduration=1.713192119 podStartE2EDuration="1.713192119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:09.709366044 +0000 UTC m=+1443.732969945" watchObservedRunningTime="2026-10-30 19:41:09.713192119 +0000 UTC m=+1443.736796018" Oct 30 19:41:09.803473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.802786 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:09.803473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.802825 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4vr\" (UniqueName: \"kubernetes.io/projected/6a662714-b2ec-421e-83ec-ca6ad917c8f8-kube-api-access-rt4vr\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:09.803473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.802840 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a662714-b2ec-421e-83ec-ca6ad917c8f8-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:09.803473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.802853 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a662714-b2ec-421e-83ec-ca6ad917c8f8-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:09.874864 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:09.874685 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:10.009959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.009920 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg"] Oct 30 19:41:10.016224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.016182 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-22af66-6c748b7445-vc9sg"] Oct 30 19:41:10.098688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.098653 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a662714-b2ec-421e-83ec-ca6ad917c8f8" path="/var/lib/kubelet/pods/6a662714-b2ec-421e-83ec-ca6ad917c8f8/volumes" Oct 30 19:41:10.264496 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.245540 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:41:10.576439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.575539 6511 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:41:10.576439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.575603 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 19:41:10.876540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.876060 6511 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:41:10.876540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:10.876125 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 19:41:11.575170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:11.575137 6511 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:41:11.575643 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:11.575592 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 19:41:11.874466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:11.874335 6511 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:41:11.874466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:11.874395 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 19:41:12.579651 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:12.579618 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:41:12.580879 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:12.580857 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:41:12.874294 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:12.874211 6511 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:41:12.874294 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:12.874263 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 19:41:13.875019 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:13.874985 6511 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:41:13.875475 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:13.875045 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 19:41:14.874940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:14.874904 6511 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:41:14.875142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:14.874969 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 19:41:15.874226 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:15.874195 6511 patch_prober.go:29] interesting pod/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:41:15.874406 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:15.874249 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 19:41:16.066669 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.066635 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb"] Oct 30 19:41:16.068152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.068126 6511 cpu_manager.go:414] "containerMap: removing container" podUID="6a662714-b2ec-421e-83ec-ca6ad917c8f8" containerName="maas-api" Oct 30 19:41:16.068302 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.068287 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a662714-b2ec-421e-83ec-ca6ad917c8f8" containerName="maas-api" Oct 30 19:41:16.068542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.068526 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a662714-b2ec-421e-83ec-ca6ad917c8f8" containerName="maas-api" Oct 30 19:41:16.074081 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.074061 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.083463 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.083405 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-47110f-serving-cert\"" Oct 30 19:41:16.100074 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.100051 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb"] Oct 30 19:41:16.153956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.153865 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.154128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.154005 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xht5r\" (UniqueName: \"kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.154128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.154077 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.154301 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.154276 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.256592 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.256501 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xht5r\" (UniqueName: \"kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.257251 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.257221 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.258244 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.258138 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.258550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.258527 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.259171 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.259152 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.271908 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.271734 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.272599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.272565 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xht5r\" (UniqueName: \"kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.273563 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.273539 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs\") pod \"maas-api-e2e-inf-b-47110f-9945c9b5d-znctb\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.389783 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.389753 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:16.727766 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.727548 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb"] Oct 30 19:41:16.786458 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.783334 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" event={"ID":"4a8e2f87-4f63-48c1-af13-b502e7fef285","Type":"ContainerStarted","Data":"5261f784297babe47d7f3d3bbea01f5baf5c5f985bbbc127f3b685be6e56ff2b"} Oct 30 19:41:16.880033 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.879998 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:16.880327 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.880309 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:16.881168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:16.881145 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:41:17.791819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:17.791786 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" event={"ID":"4a8e2f87-4f63-48c1-af13-b502e7fef285","Type":"ContainerStarted","Data":"a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803"} Oct 30 19:41:17.792217 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:17.791860 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:17.839379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:17.839323 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" podStartSLOduration=1.428955948 podStartE2EDuration="1.839309433s" totalImagesPullingTime="410.353485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.837261318 +0000 UTC m=+1451.860865229" watchObservedRunningTime="2026-10-30 19:41:17.839309433 +0000 UTC m=+1451.862913326" Oct 30 19:41:23.854121 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:23.854086 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:41:24.380011 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:41:24.379976 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:41:43.519432 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:43.519356 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7"] Oct 30 19:41:43.520087 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:43.519631 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" containerID="cri-o://4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236" gracePeriod=30 Oct 30 19:41:48.784380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.784313 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:41:48.902377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902337 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902380 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902401 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902484 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902547 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902579 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902730 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902770 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.902862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902779 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:48.902862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902829 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data\") pod \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\" (UID: \"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740\") " Oct 30 19:41:48.903079 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.902921 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:48.903192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903167 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:48.903324 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903030 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:48.903587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903331 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:48.903587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903500 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.903587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903514 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.903587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903524 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.903587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903533 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.903587 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.903541 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:48.905080 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.905058 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:48.905186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.905135 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:48.905311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.905294 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:48.905352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:48.905317 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh" pod "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" (UID: "20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740"). InnerVolumeSpecName "kube-api-access-67bxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:49.004788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.004751 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.004788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.004781 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.004788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.004791 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bxh\" (UniqueName: \"kubernetes.io/projected/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-kube-api-access-67bxh\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.005027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.004800 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:49.063758 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.063722 6511 generic.go:356] "Generic (PLEG): container finished" podID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerID="4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236" exitCode=0 Oct 30 19:41:49.063940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.063770 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" Oct 30 19:41:49.063940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.063811 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" event={"ID":"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740","Type":"ContainerDied","Data":"4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236"} Oct 30 19:41:49.063940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.063876 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7" event={"ID":"20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740","Type":"ContainerDied","Data":"51cf24e014213355bbe5c239808798308f811246b367137c8fcc362ffb244c3c"} Oct 30 19:41:49.063940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.063901 6511 scope.go:122] "RemoveContainer" containerID="4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236" Oct 30 19:41:49.074785 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.074767 6511 scope.go:122] "RemoveContainer" containerID="4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236" Oct 30 19:41:49.075053 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:41:49.075030 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236\": container with ID starting with 4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236 not found: ID does not exist" containerID="4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236" Oct 30 19:41:49.075154 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.075060 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236"} err="failed to get container status \"4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236\": rpc error: code = NotFound desc = could not find container \"4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236\": container with ID starting with 4031c8ba9c0b98f39d45d2ebd8909f0a4868a0c77445f79199c42191e87f3236 not found: ID does not exist" Oct 30 19:41:49.088543 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.088512 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7"] Oct 30 19:41:49.094444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:49.094401 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-jssx7"] Oct 30 19:41:50.099485 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:50.099442 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" path="/var/lib/kubelet/pods/20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740/volumes" Oct 30 19:41:54.379373 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:41:54.379336 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:41:59.836195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:59.836142 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t"] Oct 30 19:41:59.836762 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:41:59.836430 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" podUID="ccc436ab-c528-448a-8e43-183b46fffc3d" containerName="maas-api" containerID="cri-o://3533be782ffc598d71b5679c375d55e6db43ff31ff411a34856e88af25c52353" gracePeriod=30 Oct 30 19:42:00.270940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.268875 6511 generic.go:356] "Generic (PLEG): container finished" podID="ccc436ab-c528-448a-8e43-183b46fffc3d" containerID="3533be782ffc598d71b5679c375d55e6db43ff31ff411a34856e88af25c52353" exitCode=0 Oct 30 19:42:00.270940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.269085 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" event={"ID":"ccc436ab-c528-448a-8e43-183b46fffc3d","Type":"ContainerDied","Data":"3533be782ffc598d71b5679c375d55e6db43ff31ff411a34856e88af25c52353"} Oct 30 19:42:00.270940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.269126 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" event={"ID":"ccc436ab-c528-448a-8e43-183b46fffc3d","Type":"ContainerDied","Data":"6ad0fa02c44106291b17158897c98f2a216f1e5684fb2ad6112fdcc6577074b3"} Oct 30 19:42:00.270940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.269149 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ad0fa02c44106291b17158897c98f2a216f1e5684fb2ad6112fdcc6577074b3" Oct 30 19:42:00.270940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.270736 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:42:00.372673 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.372644 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca\") pod \"ccc436ab-c528-448a-8e43-183b46fffc3d\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " Oct 30 19:42:00.372858 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.372690 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z\" (UniqueName: \"kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z\") pod \"ccc436ab-c528-448a-8e43-183b46fffc3d\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " Oct 30 19:42:00.372925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.372868 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs\") pod \"ccc436ab-c528-448a-8e43-183b46fffc3d\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " Oct 30 19:42:00.373042 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.373028 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\") pod \"ccc436ab-c528-448a-8e43-183b46fffc3d\" (UID: \"ccc436ab-c528-448a-8e43-183b46fffc3d\") " Oct 30 19:42:00.373108 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.373079 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca" pod "ccc436ab-c528-448a-8e43-183b46fffc3d" (UID: "ccc436ab-c528-448a-8e43-183b46fffc3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:00.373534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.373511 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc436ab-c528-448a-8e43-183b46fffc3d-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:00.375005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.374983 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z" pod "ccc436ab-c528-448a-8e43-183b46fffc3d" (UID: "ccc436ab-c528-448a-8e43-183b46fffc3d"). InnerVolumeSpecName "kube-api-access-p4t4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:00.375005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.374994 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs" pod "ccc436ab-c528-448a-8e43-183b46fffc3d" (UID: "ccc436ab-c528-448a-8e43-183b46fffc3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:00.375289 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.375265 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls" pod "ccc436ab-c528-448a-8e43-183b46fffc3d" (UID: "ccc436ab-c528-448a-8e43-183b46fffc3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:00.475152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.475071 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:00.475152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.475099 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4t4z\" (UniqueName: \"kubernetes.io/projected/ccc436ab-c528-448a-8e43-183b46fffc3d-kube-api-access-p4t4z\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:00.475152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:00.475117 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc436ab-c528-448a-8e43-183b46fffc3d-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:01.276128 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:01.276098 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t" Oct 30 19:42:01.302364 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:01.302332 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t"] Oct 30 19:42:01.305046 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:01.305021 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ca9d14-68b959f579-g5v2t"] Oct 30 19:42:02.097992 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:02.097958 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc436ab-c528-448a-8e43-183b46fffc3d" path="/var/lib/kubelet/pods/ccc436ab-c528-448a-8e43-183b46fffc3d/volumes" Oct 30 19:42:06.526027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:06.526000 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:42:06.541209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:06.541187 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:42:08.416607 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:08.416580 6511 scope.go:122] "RemoveContainer" containerID="3533be782ffc598d71b5679c375d55e6db43ff31ff411a34856e88af25c52353" Oct 30 19:42:13.373312 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:13.373277 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb"] Oct 30 19:42:13.379321 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:13.379297 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7"] Oct 30 19:42:24.378776 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:42:24.378742 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:42:28.115653 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.115618 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx"] Oct 30 19:42:28.116153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.115847 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" containerID="cri-o://f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55" gracePeriod=30 Oct 30 19:42:28.395122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.395041 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" containerID="cri-o://03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5" gracePeriod=45 Oct 30 19:42:28.400102 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.399912 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" containerID="cri-o://45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d" gracePeriod=45 Oct 30 19:42:28.728204 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.728179 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:42:28.743677 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.743657 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:42:28.859639 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859605 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home\") pod \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " Oct 30 19:42:28.859639 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859652 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p\" (UniqueName: \"kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p\") pod \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " Oct 30 19:42:28.859910 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859691 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm\") pod \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " Oct 30 19:42:28.859910 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859756 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs\") pod \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " Oct 30 19:42:28.859910 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859797 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache\") pod \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " Oct 30 19:42:28.859910 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859842 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467\" (UniqueName: \"kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467\") pod \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " Oct 30 19:42:28.859910 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859894 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs\") pod \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " Oct 30 19:42:28.860173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.859940 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm\") pod \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " Oct 30 19:42:28.860173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860057 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache\") pod \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " Oct 30 19:42:28.860173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860084 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir\") pod \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\" (UID: \"dbbdf98f-0862-490f-b7f0-2357fc8782e1\") " Oct 30 19:42:28.860173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860141 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir\") pod \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " Oct 30 19:42:28.860374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860238 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home\") pod \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\" (UID: \"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7\") " Oct 30 19:42:28.860524 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860498 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache" pod "dbbdf98f-0862-490f-b7f0-2357fc8782e1" (UID: "dbbdf98f-0862-490f-b7f0-2357fc8782e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.860688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860667 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache" pod "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" (UID: "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.860842 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860788 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir" pod "dbbdf98f-0862-490f-b7f0-2357fc8782e1" (UID: "dbbdf98f-0862-490f-b7f0-2357fc8782e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.860947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.860845 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home" pod "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" (UID: "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.861050 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.861032 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.861115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.861062 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.861115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.861078 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.861115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.861092 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.861115 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.861030 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir" pod "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" (UID: "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.861317 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.861167 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home" pod "dbbdf98f-0862-490f-b7f0-2357fc8782e1" (UID: "dbbdf98f-0862-490f-b7f0-2357fc8782e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.864791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.864750 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs" pod "dbbdf98f-0862-490f-b7f0-2357fc8782e1" (UID: "dbbdf98f-0862-490f-b7f0-2357fc8782e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:28.864791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.864760 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm" pod "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" (UID: "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.864791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.864782 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs" pod "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" (UID: "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:28.865041 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.864795 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p" pod "dbbdf98f-0862-490f-b7f0-2357fc8782e1" (UID: "dbbdf98f-0862-490f-b7f0-2357fc8782e1"). InnerVolumeSpecName "kube-api-access-9s76p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:28.865195 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.865166 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm" pod "dbbdf98f-0862-490f-b7f0-2357fc8782e1" (UID: "dbbdf98f-0862-490f-b7f0-2357fc8782e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:28.866191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.866165 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467" pod "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" (UID: "e0ef7861-ba7f-4358-8f3b-ffeaca572ee7"). InnerVolumeSpecName "kube-api-access-6w467". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962831 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbbdf98f-0862-490f-b7f0-2357fc8782e1-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962860 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w467\" (UniqueName: \"kubernetes.io/projected/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-kube-api-access-6w467\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962873 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962882 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962894 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962905 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbbdf98f-0862-490f-b7f0-2357fc8782e1-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.962912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962917 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s76p\" (UniqueName: \"kubernetes.io/projected/dbbdf98f-0862-490f-b7f0-2357fc8782e1-kube-api-access-9s76p\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:28.963452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:28.962929 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:29.504142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.504101 6511 generic.go:356] "Generic (PLEG): container finished" podID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerID="03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5" exitCode=0 Oct 30 19:42:29.504631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.504143 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" event={"ID":"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7","Type":"ContainerDied","Data":"03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5"} Oct 30 19:42:29.504631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.504192 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" event={"ID":"e0ef7861-ba7f-4358-8f3b-ffeaca572ee7","Type":"ContainerDied","Data":"a4a9816b2a9a7a55c3a2e23cb5c344333d91da415884b496b38383540a3279b0"} Oct 30 19:42:29.504631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.504212 6511 scope.go:122] "RemoveContainer" containerID="03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5" Oct 30 19:42:29.504631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.504150 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" Oct 30 19:42:29.505752 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.505733 6511 generic.go:356] "Generic (PLEG): container finished" podID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerID="45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d" exitCode=0 Oct 30 19:42:29.505845 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.505773 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" Oct 30 19:42:29.505909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.505853 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" event={"ID":"dbbdf98f-0862-490f-b7f0-2357fc8782e1","Type":"ContainerDied","Data":"45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d"} Oct 30 19:42:29.505909 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.505876 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7" event={"ID":"dbbdf98f-0862-490f-b7f0-2357fc8782e1","Type":"ContainerDied","Data":"5409afe8ec714187c9bb5e65bbdd302bdfb4eff21dc64a619ec1fc712e9888a2"} Oct 30 19:42:29.514829 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.514804 6511 scope.go:122] "RemoveContainer" containerID="03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5" Oct 30 19:42:29.523572 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:42:29.519725 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5\": container with ID starting with 03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5 not found: ID does not exist" containerID="03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5" Oct 30 19:42:29.523572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.519764 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5"} err="failed to get container status \"03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5\": rpc error: code = NotFound desc = could not find container \"03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5\": container with ID starting with 03f18e5d494e7e87976ffb78f9a49f57f391af20209aa936735e9c005eb4dfc5 not found: ID does not exist" Oct 30 19:42:29.523572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.519789 6511 scope.go:122] "RemoveContainer" containerID="45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d" Oct 30 19:42:29.530968 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.530938 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb"] Oct 30 19:42:29.532993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.532974 6511 scope.go:122] "RemoveContainer" containerID="45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d" Oct 30 19:42:29.533277 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:42:29.533259 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d\": container with ID starting with 45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d not found: ID does not exist" containerID="45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d" Oct 30 19:42:29.533337 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.533284 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d"} err="failed to get container status \"45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d\": rpc error: code = NotFound desc = could not find container \"45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d\": container with ID starting with 45a09216afc4e2420f7ea9f646b08d7e39212b26b752d9eb41a5230191fbf32d not found: ID does not exist" Oct 30 19:42:29.535956 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.535937 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb"] Oct 30 19:42:29.548291 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.548258 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7"] Oct 30 19:42:29.550782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.550763 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b623ca/distinct-w2-b623ca-kserve-c4559454d-kg5c7"] Oct 30 19:42:29.665099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:29.665013 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b623ca/distinct-2-w2-b623ca-kserve-995469d67-gl6tb" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:30.108848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:30.108811 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" path="/var/lib/kubelet/pods/dbbdf98f-0862-490f-b7f0-2357fc8782e1/volumes" Oct 30 19:42:30.109797 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:30.109771 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" path="/var/lib/kubelet/pods/e0ef7861-ba7f-4358-8f3b-ffeaca572ee7/volumes" Oct 30 19:42:33.407200 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.407173 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:42:33.522116 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522084 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522200 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522219 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522282 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522311 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522304 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522345 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522404 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522454 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:33.522637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522505 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522567 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo\") pod \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\" (UID: \"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448\") " Oct 30 19:42:33.522903 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522670 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:33.522903 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522700 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:33.522903 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522785 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:33.522903 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.522892 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:33.523175 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.523156 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.523243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.523182 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.523243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.523197 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.523243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.523213 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.523243 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.523228 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.524632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.524606 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:33.524799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.524726 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:33.524863 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.524839 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:33.525091 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.525073 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n" pod "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" (UID: "11d505aa-9ea6-4e1a-b3b8-d9da39b8b448"). InnerVolumeSpecName "kube-api-access-7n69n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:33.533314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.533139 6511 generic.go:356] "Generic (PLEG): container finished" podID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerID="f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55" exitCode=0 Oct 30 19:42:33.533314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.533200 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" Oct 30 19:42:33.533314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.533198 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" event={"ID":"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448","Type":"ContainerDied","Data":"f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55"} Oct 30 19:42:33.533314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.533268 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx" event={"ID":"11d505aa-9ea6-4e1a-b3b8-d9da39b8b448","Type":"ContainerDied","Data":"5f3fe9f7e68869ff4a156a6fa5bc2fbe3a7a7aa754f3f7b7bf21e7f6d280d9ff"} Oct 30 19:42:33.533314 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.533298 6511 scope.go:122] "RemoveContainer" containerID="f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55" Oct 30 19:42:33.556764 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.556742 6511 scope.go:122] "RemoveContainer" containerID="f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55" Oct 30 19:42:33.557115 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:42:33.557092 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55\": container with ID starting with f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55 not found: ID does not exist" containerID="f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55" Oct 30 19:42:33.557205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.557123 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55"} err="failed to get container status \"f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55\": rpc error: code = NotFound desc = could not find container \"f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55\": container with ID starting with f024f74a0311b7177ba55ce2620a23c30892d83eedd217529c3d30f145637c55 not found: ID does not exist" Oct 30 19:42:33.570027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.570001 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx"] Oct 30 19:42:33.572734 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.572710 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-bdpxx"] Oct 30 19:42:33.624501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.624476 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.624501 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.624499 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.624674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.624514 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:33.624674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:33.624523 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n69n\" (UniqueName: \"kubernetes.io/projected/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448-kube-api-access-7n69n\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:34.126470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:34.125275 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" path="/var/lib/kubelet/pods/11d505aa-9ea6-4e1a-b3b8-d9da39b8b448/volumes" Oct 30 19:42:43.067745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.067714 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r"] Oct 30 19:42:43.068303 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068287 6511 cpu_manager.go:414] "containerMap: removing container" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" Oct 30 19:42:43.068353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068306 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" Oct 30 19:42:43.068353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068322 6511 cpu_manager.go:414] "containerMap: removing container" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" Oct 30 19:42:43.068353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068327 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" Oct 30 19:42:43.068353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068340 6511 cpu_manager.go:414] "containerMap: removing container" podUID="ccc436ab-c528-448a-8e43-183b46fffc3d" containerName="maas-api" Oct 30 19:42:43.068353 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068349 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc436ab-c528-448a-8e43-183b46fffc3d" containerName="maas-api" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068379 6511 cpu_manager.go:414] "containerMap: removing container" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068384 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068393 6511 cpu_manager.go:414] "containerMap: removing container" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068397 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068494 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d505aa-9ea6-4e1a-b3b8-d9da39b8b448" containerName="istio-proxy" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068503 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbbdf98f-0862-490f-b7f0-2357fc8782e1" containerName="main" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068512 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f8cc8b-d0bc-4a3f-8e75-e1e5b9604740" containerName="istio-proxy" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068519 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ef7861-ba7f-4358-8f3b-ffeaca572ee7" containerName="main" Oct 30 19:42:43.068579 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.068524 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc436ab-c528-448a-8e43-183b46fffc3d" containerName="maas-api" Oct 30 19:42:43.073371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.073356 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.082683 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.082661 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-47110f-openshift-default-dockercfg-vqr44\"" Oct 30 19:42:43.086362 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.086334 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r"] Oct 30 19:42:43.147946 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.147919 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148099 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.147966 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148160 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148115 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148199 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148168 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148253 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148238 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148289 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148264 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzj6x\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148339 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148294 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148430 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148402 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.148527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.148485 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249277 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249325 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249345 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249362 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzj6x\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249390 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249472 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249518 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249624 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249677 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.249780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249762 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.250209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249781 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.250209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249966 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.250209 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.249992 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.250489 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.250465 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.252072 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.252048 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.252326 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.252306 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.257660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.257634 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.257867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.257849 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzj6x\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x\") pod \"e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.388970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.388898 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:43.677684 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.677640 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r"] Oct 30 19:42:43.677824 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:42:43.677638 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6c6b51_c969_451b_b1fe_6287758d2679.slice/crio-29722d318471a91e3804139578806f154a9634d6375e1f6ff6ca065a28f00022 WatchSource:0}: Error finding container 29722d318471a91e3804139578806f154a9634d6375e1f6ff6ca065a28f00022: Status 404 returned error can't find the container with id 29722d318471a91e3804139578806f154a9634d6375e1f6ff6ca065a28f00022 Oct 30 19:42:43.680608 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.680566 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:42:43.680714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.680649 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:42:43.680714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:43.680690 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:42:44.380374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.380343 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5"] Oct 30 19:42:44.381027 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.380975 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" podUID="bdd6751c-1408-4557-82bc-d9e581864563" containerName="maas-api" containerID="cri-o://a5db530d0a4d42f0a00f24e025907010784c00b3b38e9c539535f1a2a8eed5a7" gracePeriod=30 Oct 30 19:42:44.740444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.739532 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" event={"ID":"1b6c6b51-c969-451b-b1fe-6287758d2679","Type":"ContainerStarted","Data":"7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0"} Oct 30 19:42:44.740444 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.739602 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" event={"ID":"1b6c6b51-c969-451b-b1fe-6287758d2679","Type":"ContainerStarted","Data":"29722d318471a91e3804139578806f154a9634d6375e1f6ff6ca065a28f00022"} Oct 30 19:42:44.810437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.803443 6511 generic.go:356] "Generic (PLEG): container finished" podID="bdd6751c-1408-4557-82bc-d9e581864563" containerID="a5db530d0a4d42f0a00f24e025907010784c00b3b38e9c539535f1a2a8eed5a7" exitCode=0 Oct 30 19:42:44.810437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.803588 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" event={"ID":"bdd6751c-1408-4557-82bc-d9e581864563","Type":"ContainerDied","Data":"a5db530d0a4d42f0a00f24e025907010784c00b3b38e9c539535f1a2a8eed5a7"} Oct 30 19:42:44.858439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:44.857626 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podStartSLOduration=1.857604569 podStartE2EDuration="1.857604569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:44.804084369 +0000 UTC m=+1538.827688269" watchObservedRunningTime="2026-10-30 19:42:44.857604569 +0000 UTC m=+1538.881208471" Oct 30 19:42:45.005937 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.005901 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:42:45.154637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.154598 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca\") pod \"bdd6751c-1408-4557-82bc-d9e581864563\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " Oct 30 19:42:45.154831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.154653 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs\") pod \"bdd6751c-1408-4557-82bc-d9e581864563\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " Oct 30 19:42:45.154831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.154716 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls\") pod \"bdd6751c-1408-4557-82bc-d9e581864563\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " Oct 30 19:42:45.154831 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.154825 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx\" (UniqueName: \"kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx\") pod \"bdd6751c-1408-4557-82bc-d9e581864563\" (UID: \"bdd6751c-1408-4557-82bc-d9e581864563\") " Oct 30 19:42:45.155025 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.154999 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca" pod "bdd6751c-1408-4557-82bc-d9e581864563" (UID: "bdd6751c-1408-4557-82bc-d9e581864563"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:45.155317 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.155289 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd6751c-1408-4557-82bc-d9e581864563-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:45.157268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.157242 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx" pod "bdd6751c-1408-4557-82bc-d9e581864563" (UID: "bdd6751c-1408-4557-82bc-d9e581864563"). InnerVolumeSpecName "kube-api-access-sqncx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:45.157443 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.157395 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls" pod "bdd6751c-1408-4557-82bc-d9e581864563" (UID: "bdd6751c-1408-4557-82bc-d9e581864563"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:45.157523 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.157486 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs" pod "bdd6751c-1408-4557-82bc-d9e581864563" (UID: "bdd6751c-1408-4557-82bc-d9e581864563"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:45.256605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.256529 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:45.256605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.256555 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqncx\" (UniqueName: \"kubernetes.io/projected/bdd6751c-1408-4557-82bc-d9e581864563-kube-api-access-sqncx\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:45.256605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.256566 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd6751c-1408-4557-82bc-d9e581864563-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:45.389529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.389499 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:45.390894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.390870 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:42:45.391017 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.390927 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:42:45.812225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.812191 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" Oct 30 19:42:45.812371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.812215 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5" event={"ID":"bdd6751c-1408-4557-82bc-d9e581864563","Type":"ContainerDied","Data":"e2386dc2b8fc953da6024d3abcbb7525ef0cf508544f276111f4a7ecac3963bd"} Oct 30 19:42:45.812371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.812277 6511 scope.go:122] "RemoveContainer" containerID="a5db530d0a4d42f0a00f24e025907010784c00b3b38e9c539535f1a2a8eed5a7" Oct 30 19:42:45.869967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.869934 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5"] Oct 30 19:42:45.872676 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:45.872655 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b623ca-bb7547596-fxrz5"] Oct 30 19:42:46.097295 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:46.097221 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd6751c-1408-4557-82bc-d9e581864563" path="/var/lib/kubelet/pods/bdd6751c-1408-4557-82bc-d9e581864563/volumes" Oct 30 19:42:46.390106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:46.390026 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:42:46.390106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:46.390070 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:42:47.389551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:47.389513 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:42:47.389723 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:47.389581 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:42:48.389620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:48.389577 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:42:48.390021 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:48.389634 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:42:49.389333 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:49.389287 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:42:49.389540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:49.389353 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:42:50.390101 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:50.390053 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:42:50.390605 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:50.390134 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:42:51.396275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:51.396243 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:51.396797 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:51.396750 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:51.401639 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:51.401618 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:42:54.021116 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:54.021067 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx"] Oct 30 19:42:54.021742 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:54.021307 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" containerID="cri-o://1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec" gracePeriod=30 Oct 30 19:42:54.028214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:54.028189 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr"] Oct 30 19:42:54.028544 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:54.028391 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" containerID="cri-o://d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda" gracePeriod=30 Oct 30 19:42:54.389890 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:42:54.389853 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:42:59.314931 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.314904 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:42:59.336854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.336832 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:42:59.382225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382185 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382425 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382239 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382425 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382270 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382425 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382348 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382425 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382383 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382458 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382488 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382526 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382597 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382692 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382647 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382697 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.382967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382747 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382808 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382852 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382888 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.382967 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.382914 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token\") pod \"5abf50bf-3416-46b3-b084-796963443d56\" (UID: \"5abf50bf-3416-46b3-b084-796963443d56\") " Oct 30 19:42:59.384701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.383627 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.384701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.383703 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8\") pod \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\" (UID: \"d20f0cec-20fa-4300-8838-18f0ef03e9dc\") " Oct 30 19:42:59.385298 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385207 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.385298 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385222 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.385495 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385304 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385558 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385567 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385678 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385884 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.385997 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.386061 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.386280 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.386547 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.386496 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.386987 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.386788 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "kube-api-access-dpbv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.386987 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.386957 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.387334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.387229 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.387874 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.387853 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo" pod "d20f0cec-20fa-4300-8838-18f0ef03e9dc" (UID: "d20f0cec-20fa-4300-8838-18f0ef03e9dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:59.388293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.388277 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:59.389182 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.389163 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "kube-api-access-sp7tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:59.389465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.389434 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy" pod "5abf50bf-3416-46b3-b084-796963443d56" (UID: "5abf50bf-3416-46b3-b084-796963443d56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:59.485672 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485633 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485676 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485693 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5abf50bf-3416-46b3-b084-796963443d56-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485708 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485725 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485739 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485754 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485768 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485781 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbv8\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-kube-api-access-dpbv8\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485798 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485812 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.485828 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485825 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5abf50bf-3416-46b3-b084-796963443d56-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.486170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485840 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp7tp\" (UniqueName: \"kubernetes.io/projected/5abf50bf-3416-46b3-b084-796963443d56-kube-api-access-sp7tp\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.486170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485854 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.486170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485869 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.486170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485883 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.486170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485896 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5abf50bf-3416-46b3-b084-796963443d56-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:59.486170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:42:59.485910 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20f0cec-20fa-4300-8838-18f0ef03e9dc-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:00.024156 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.024118 6511 generic.go:356] "Generic (PLEG): container finished" podID="5abf50bf-3416-46b3-b084-796963443d56" containerID="d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda" exitCode=0 Oct 30 19:43:00.024347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.024172 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" Oct 30 19:43:00.024347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.024200 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" event={"ID":"5abf50bf-3416-46b3-b084-796963443d56","Type":"ContainerDied","Data":"d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda"} Oct 30 19:43:00.024347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.024273 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr" event={"ID":"5abf50bf-3416-46b3-b084-796963443d56","Type":"ContainerDied","Data":"94f71f12f8c26643f568a4e1b8644cb204910faff4e475f047e7b93e54edc956"} Oct 30 19:43:00.024347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.024307 6511 scope.go:122] "RemoveContainer" containerID="d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda" Oct 30 19:43:00.025932 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.025907 6511 generic.go:356] "Generic (PLEG): container finished" podID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerID="1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec" exitCode=0 Oct 30 19:43:00.026153 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.026136 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" event={"ID":"d20f0cec-20fa-4300-8838-18f0ef03e9dc","Type":"ContainerDied","Data":"1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec"} Oct 30 19:43:00.026228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.026185 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" event={"ID":"d20f0cec-20fa-4300-8838-18f0ef03e9dc","Type":"ContainerDied","Data":"a62bb7a4d17a031b0b3924442f60f4553e00b2117303c5ffda77c7cbdcb3773f"} Oct 30 19:43:00.026278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.026259 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx" Oct 30 19:43:00.034275 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.034259 6511 scope.go:122] "RemoveContainer" containerID="d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda" Oct 30 19:43:00.034553 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:43:00.034528 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda\": container with ID starting with d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda not found: ID does not exist" containerID="d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda" Oct 30 19:43:00.034646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.034562 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda"} err="failed to get container status \"d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda\": rpc error: code = NotFound desc = could not find container \"d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda\": container with ID starting with d19b80c1419e097eef07c63cd666c77d2c379d552cc5fb6253ccb1dbe9eb5cda not found: ID does not exist" Oct 30 19:43:00.034646 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.034585 6511 scope.go:122] "RemoveContainer" containerID="1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec" Oct 30 19:43:00.044272 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.044254 6511 scope.go:122] "RemoveContainer" containerID="1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec" Oct 30 19:43:00.044604 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:43:00.044583 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec\": container with ID starting with 1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec not found: ID does not exist" containerID="1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec" Oct 30 19:43:00.044695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.044608 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec"} err="failed to get container status \"1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec\": rpc error: code = NotFound desc = could not find container \"1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec\": container with ID starting with 1df446b151469e611d224f4b48a951d09bc944233ce15f61994d086d0b59d7ec not found: ID does not exist" Oct 30 19:43:00.055151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.055121 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr"] Oct 30 19:43:00.059466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.059443 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-bsbrr"] Oct 30 19:43:00.070924 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.070906 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx"] Oct 30 19:43:00.076427 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.076394 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b623ca-openshift-default-68847758ff-5dpgx"] Oct 30 19:43:00.097148 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.097123 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abf50bf-3416-46b3-b084-796963443d56" path="/var/lib/kubelet/pods/5abf50bf-3416-46b3-b084-796963443d56/volumes" Oct 30 19:43:00.097628 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:00.097615 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" path="/var/lib/kubelet/pods/d20f0cec-20fa-4300-8838-18f0ef03e9dc/volumes" Oct 30 19:43:24.419638 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:43:24.418050 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:43:42.373282 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.373244 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp"] Oct 30 19:43:42.373781 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.373518 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" podUID="aa7c687e-86c8-4047-844e-ca762470c1ba" containerName="maas-api" containerID="cri-o://2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e" gracePeriod=30 Oct 30 19:43:42.767227 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.767197 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:43:42.881588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.881545 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca\") pod \"aa7c687e-86c8-4047-844e-ca762470c1ba\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " Oct 30 19:43:42.881776 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.881689 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls\") pod \"aa7c687e-86c8-4047-844e-ca762470c1ba\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " Oct 30 19:43:42.881875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.881860 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6\" (UniqueName: \"kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6\") pod \"aa7c687e-86c8-4047-844e-ca762470c1ba\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " Oct 30 19:43:42.881990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.881963 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca" pod "aa7c687e-86c8-4047-844e-ca762470c1ba" (UID: "aa7c687e-86c8-4047-844e-ca762470c1ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:42.882084 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.881894 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs\") pod \"aa7c687e-86c8-4047-844e-ca762470c1ba\" (UID: \"aa7c687e-86c8-4047-844e-ca762470c1ba\") " Oct 30 19:43:42.882447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.882392 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7c687e-86c8-4047-844e-ca762470c1ba-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:42.884211 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.884183 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6" pod "aa7c687e-86c8-4047-844e-ca762470c1ba" (UID: "aa7c687e-86c8-4047-844e-ca762470c1ba"). InnerVolumeSpecName "kube-api-access-fvqk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:42.884535 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.884520 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs" pod "aa7c687e-86c8-4047-844e-ca762470c1ba" (UID: "aa7c687e-86c8-4047-844e-ca762470c1ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:42.884593 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.884568 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls" pod "aa7c687e-86c8-4047-844e-ca762470c1ba" (UID: "aa7c687e-86c8-4047-844e-ca762470c1ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:42.983796 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.983722 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:42.983796 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.983762 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvqk6\" (UniqueName: \"kubernetes.io/projected/aa7c687e-86c8-4047-844e-ca762470c1ba-kube-api-access-fvqk6\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:42.983796 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:42.983773 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7c687e-86c8-4047-844e-ca762470c1ba-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:43.341013 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.340977 6511 generic.go:356] "Generic (PLEG): container finished" podID="aa7c687e-86c8-4047-844e-ca762470c1ba" containerID="2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e" exitCode=0 Oct 30 19:43:43.341191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.341032 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" Oct 30 19:43:43.341191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.341034 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" event={"ID":"aa7c687e-86c8-4047-844e-ca762470c1ba","Type":"ContainerDied","Data":"2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e"} Oct 30 19:43:43.341191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.341155 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp" event={"ID":"aa7c687e-86c8-4047-844e-ca762470c1ba","Type":"ContainerDied","Data":"c408f65484c157b13230dcb28f2b8a8b62a92d5d10027ff71c651d3dde648315"} Oct 30 19:43:43.341191 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.341176 6511 scope.go:122] "RemoveContainer" containerID="2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e" Oct 30 19:43:43.351460 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.351440 6511 scope.go:122] "RemoveContainer" containerID="2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e" Oct 30 19:43:43.351746 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:43:43.351724 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e\": container with ID starting with 2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e not found: ID does not exist" containerID="2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e" Oct 30 19:43:43.351822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.351756 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e"} err="failed to get container status \"2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e\": rpc error: code = NotFound desc = could not find container \"2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e\": container with ID starting with 2c0b0df6da79cbdef373daaae579cf92d1f45bc148d252cc07b2f0029de5918e not found: ID does not exist" Oct 30 19:43:43.366603 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.366582 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp"] Oct 30 19:43:43.374802 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:43.374782 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-579ddb-68b57f8859-h7qlp"] Oct 30 19:43:44.097003 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:43:44.096962 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7c687e-86c8-4047-844e-ca762470c1ba" path="/var/lib/kubelet/pods/aa7c687e-86c8-4047-844e-ca762470c1ba/volumes" Oct 30 19:43:54.378495 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:43:54.378449 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:44:06.140081 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.140047 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98"] Oct 30 19:44:06.140570 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.140284 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" podUID="606e46d4-427c-4d9b-b63f-efecaa8d0849" containerName="maas-api" containerID="cri-o://d5d3c91fa965db11e88be1496989f4bc9a8387bf0bccadba910957287f5ee857" gracePeriod=30 Oct 30 19:44:06.483705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.483666 6511 generic.go:356] "Generic (PLEG): container finished" podID="606e46d4-427c-4d9b-b63f-efecaa8d0849" containerID="d5d3c91fa965db11e88be1496989f4bc9a8387bf0bccadba910957287f5ee857" exitCode=0 Oct 30 19:44:06.483896 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.483779 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" event={"ID":"606e46d4-427c-4d9b-b63f-efecaa8d0849","Type":"ContainerDied","Data":"d5d3c91fa965db11e88be1496989f4bc9a8387bf0bccadba910957287f5ee857"} Oct 30 19:44:06.644359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.644335 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:44:06.723159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.723075 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca\") pod \"606e46d4-427c-4d9b-b63f-efecaa8d0849\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " Oct 30 19:44:06.723356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.723202 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls\") pod \"606e46d4-427c-4d9b-b63f-efecaa8d0849\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " Oct 30 19:44:06.723356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.723236 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs\") pod \"606e46d4-427c-4d9b-b63f-efecaa8d0849\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " Oct 30 19:44:06.723356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.723292 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk\" (UniqueName: \"kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk\") pod \"606e46d4-427c-4d9b-b63f-efecaa8d0849\" (UID: \"606e46d4-427c-4d9b-b63f-efecaa8d0849\") " Oct 30 19:44:06.723575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.723513 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca" pod "606e46d4-427c-4d9b-b63f-efecaa8d0849" (UID: "606e46d4-427c-4d9b-b63f-efecaa8d0849"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:06.723925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.723904 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606e46d4-427c-4d9b-b63f-efecaa8d0849-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:06.725935 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.725904 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk" pod "606e46d4-427c-4d9b-b63f-efecaa8d0849" (UID: "606e46d4-427c-4d9b-b63f-efecaa8d0849"). InnerVolumeSpecName "kube-api-access-tnfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:06.726530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.726505 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls" pod "606e46d4-427c-4d9b-b63f-efecaa8d0849" (UID: "606e46d4-427c-4d9b-b63f-efecaa8d0849"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:06.726530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.726514 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs" pod "606e46d4-427c-4d9b-b63f-efecaa8d0849" (UID: "606e46d4-427c-4d9b-b63f-efecaa8d0849"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:06.825527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.825492 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:06.825527 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.825526 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606e46d4-427c-4d9b-b63f-efecaa8d0849-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:06.825745 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:06.825536 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnfmk\" (UniqueName: \"kubernetes.io/projected/606e46d4-427c-4d9b-b63f-efecaa8d0849-kube-api-access-tnfmk\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:07.490834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:07.490803 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" Oct 30 19:44:07.491222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:07.490836 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98" event={"ID":"606e46d4-427c-4d9b-b63f-efecaa8d0849","Type":"ContainerDied","Data":"cb313bff7c41c3b7f2bddc8ed0533a57a338879fa0e242726bb7b7edcb8fff7f"} Oct 30 19:44:07.491222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:07.490900 6511 scope.go:122] "RemoveContainer" containerID="d5d3c91fa965db11e88be1496989f4bc9a8387bf0bccadba910957287f5ee857" Oct 30 19:44:07.515824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:07.515799 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98"] Oct 30 19:44:07.524799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:07.524774 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-744b57-7cfc5d8cdf-bkf98"] Oct 30 19:44:08.100622 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:08.100584 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606e46d4-427c-4d9b-b63f-efecaa8d0849" path="/var/lib/kubelet/pods/606e46d4-427c-4d9b-b63f-efecaa8d0849/volumes" Oct 30 19:44:10.832473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:10.832441 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf"] Oct 30 19:44:10.832965 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:10.832680 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" podUID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" containerName="discovery" containerID="cri-o://e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc" gracePeriod=30 Oct 30 19:44:11.108514 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.108489 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:44:11.286635 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.286603 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.286788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.286667 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.286886 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.286797 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.286886 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.286845 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.286986 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.286881 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.287069 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.287040 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.287127 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.287102 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts\") pod \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\" (UID: \"4c25a6ef-b4b2-4e36-b3cc-847220439ec7\") " Oct 30 19:44:11.287575 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.287267 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:11.287871 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.287737 6511 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-ca-configmap\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.290131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.290105 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "kube-api-access-756m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:11.290131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.290123 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:11.290276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.290136 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:11.290276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.290153 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:11.290276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.290239 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:11.290276 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.290255 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token" pod "4c25a6ef-b4b2-4e36-b3cc-847220439ec7" (UID: "4c25a6ef-b4b2-4e36-b3cc-847220439ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:11.388732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.388651 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.388732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.388691 6511 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-cacerts\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.388732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.388701 6511 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-csr-dns-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.388732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.388711 6511 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-local-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.388732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.388719 6511 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-istio-kubeconfig\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.388732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.388727 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756m6\" (UniqueName: \"kubernetes.io/projected/4c25a6ef-b4b2-4e36-b3cc-847220439ec7-kube-api-access-756m6\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:11.516091 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.516050 6511 generic.go:356] "Generic (PLEG): container finished" podID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" containerID="e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc" exitCode=0 Oct 30 19:44:11.516299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.516102 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" Oct 30 19:44:11.516299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.516130 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" event={"ID":"4c25a6ef-b4b2-4e36-b3cc-847220439ec7","Type":"ContainerDied","Data":"e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc"} Oct 30 19:44:11.516299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.516200 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf" event={"ID":"4c25a6ef-b4b2-4e36-b3cc-847220439ec7","Type":"ContainerDied","Data":"f3290946c0968b7fee4dd3d75589c8415d4bc1ce47d43255e4e0c4ec2b9c9587"} Oct 30 19:44:11.516299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.516230 6511 scope.go:122] "RemoveContainer" containerID="e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc" Oct 30 19:44:11.528550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.528524 6511 scope.go:122] "RemoveContainer" containerID="e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc" Oct 30 19:44:11.528845 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:44:11.528817 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc\": container with ID starting with e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc not found: ID does not exist" containerID="e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc" Oct 30 19:44:11.528902 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.528860 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc"} err="failed to get container status \"e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc\": rpc error: code = NotFound desc = could not find container \"e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc\": container with ID starting with e6f806a2c73460d18f00c9fadd639e9cdbe3debe3a590bd5d1031cfcd8dc84fc not found: ID does not exist" Oct 30 19:44:11.553121 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.553088 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf"] Oct 30 19:44:11.566594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:11.566573 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2zxhf"] Oct 30 19:44:12.096691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:12.096655 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" path="/var/lib/kubelet/pods/4c25a6ef-b4b2-4e36-b3cc-847220439ec7/volumes" Oct 30 19:44:16.067398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.067366 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867"] Oct 30 19:44:16.068092 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.067613 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" podUID="4897e42d-9776-4111-9df9-73622a74f67c" containerName="maas-api" containerID="cri-o://c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13" gracePeriod=30 Oct 30 19:44:16.493158 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.493129 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:44:16.554521 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.554484 6511 generic.go:356] "Generic (PLEG): container finished" podID="4897e42d-9776-4111-9df9-73622a74f67c" containerID="c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13" exitCode=0 Oct 30 19:44:16.554721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.554546 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" Oct 30 19:44:16.554721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.554570 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" event={"ID":"4897e42d-9776-4111-9df9-73622a74f67c","Type":"ContainerDied","Data":"c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13"} Oct 30 19:44:16.554721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.554637 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" event={"ID":"4897e42d-9776-4111-9df9-73622a74f67c","Type":"ContainerDied","Data":"f181b2d31bc8b61c0ae6e8445c3ac9ffe8643eb9f7ba57a8d6f45c9812c8edba"} Oct 30 19:44:16.554721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.554666 6511 scope.go:122] "RemoveContainer" containerID="c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13" Oct 30 19:44:16.558599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.558575 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca\") pod \"4897e42d-9776-4111-9df9-73622a74f67c\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " Oct 30 19:44:16.558730 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.558692 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw\" (UniqueName: \"kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw\") pod \"4897e42d-9776-4111-9df9-73622a74f67c\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " Oct 30 19:44:16.558805 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.558732 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls\") pod \"4897e42d-9776-4111-9df9-73622a74f67c\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " Oct 30 19:44:16.558878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.558826 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs\") pod \"4897e42d-9776-4111-9df9-73622a74f67c\" (UID: \"4897e42d-9776-4111-9df9-73622a74f67c\") " Oct 30 19:44:16.559056 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.559023 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca" pod "4897e42d-9776-4111-9df9-73622a74f67c" (UID: "4897e42d-9776-4111-9df9-73622a74f67c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:16.559335 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.559312 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4897e42d-9776-4111-9df9-73622a74f67c-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:16.562924 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.562793 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls" pod "4897e42d-9776-4111-9df9-73622a74f67c" (UID: "4897e42d-9776-4111-9df9-73622a74f67c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:16.562924 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.562887 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw" pod "4897e42d-9776-4111-9df9-73622a74f67c" (UID: "4897e42d-9776-4111-9df9-73622a74f67c"). InnerVolumeSpecName "kube-api-access-49qpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:16.563093 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.563067 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs" pod "4897e42d-9776-4111-9df9-73622a74f67c" (UID: "4897e42d-9776-4111-9df9-73622a74f67c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:16.571131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.571053 6511 scope.go:122] "RemoveContainer" containerID="c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13" Oct 30 19:44:16.571857 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:44:16.571828 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13\": container with ID starting with c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13 not found: ID does not exist" containerID="c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13" Oct 30 19:44:16.571978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.571870 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13"} err="failed to get container status \"c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13\": rpc error: code = NotFound desc = could not find container \"c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13\": container with ID starting with c3fc5c371e78e3a450a591da37af92e6ef747c3be1376a63fb3e0ec66aa89b13 not found: ID does not exist" Oct 30 19:44:16.660728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.660702 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qpw\" (UniqueName: \"kubernetes.io/projected/4897e42d-9776-4111-9df9-73622a74f67c-kube-api-access-49qpw\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:16.660728 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.660728 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:16.660886 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.660736 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4897e42d-9776-4111-9df9-73622a74f67c-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:16.881838 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.881809 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867"] Oct 30 19:44:16.889292 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:16.889267 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867"] Oct 30 19:44:18.097737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:18.097701 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4897e42d-9776-4111-9df9-73622a74f67c" path="/var/lib/kubelet/pods/4897e42d-9776-4111-9df9-73622a74f67c/volumes" Oct 30 19:44:19.396578 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:19.396528 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1e13df-77ccfcbc9f-jp867" podUID="4897e42d-9776-4111-9df9-73622a74f67c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.93:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:24.392068 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:44:24.392028 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:44:25.490540 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:25.490505 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq"] Oct 30 19:44:25.490993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:25.490709 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" containerID="cri-o://7d53d204c90a4201870c78121c3df5969eb929a733d45f43633fceab05c9ee32" gracePeriod=30 Oct 30 19:44:30.659877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.659596 6511 generic.go:356] "Generic (PLEG): container finished" podID="c7864046-3736-42b7-9caa-2f0214187790" containerID="7d53d204c90a4201870c78121c3df5969eb929a733d45f43633fceab05c9ee32" exitCode=0 Oct 30 19:44:30.659877 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.659797 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" event={"ID":"c7864046-3736-42b7-9caa-2f0214187790","Type":"ContainerDied","Data":"7d53d204c90a4201870c78121c3df5969eb929a733d45f43633fceab05c9ee32"} Oct 30 19:44:30.744252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.744225 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:44:30.834750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.834720 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.834947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.834775 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.834947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.834811 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.834947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.834852 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.834947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.834905 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.835169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.834990 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:30.835169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835006 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.835169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835104 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:30.835169 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835145 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.835392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835237 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.835392 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835345 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data\") pod \"c7864046-3736-42b7-9caa-2f0214187790\" (UID: \"c7864046-3736-42b7-9caa-2f0214187790\") " Oct 30 19:44:30.835549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835407 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:30.835549 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835539 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:30.835660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.835635 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:30.836136 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.836114 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.836255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.836139 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.836255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.836150 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7864046-3736-42b7-9caa-2f0214187790-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.836255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.836159 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.836255 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.836168 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.837470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.837443 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:30.837660 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.837642 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:30.837878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.837848 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:30.838403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.838383 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw" pod "c7864046-3736-42b7-9caa-2f0214187790" (UID: "c7864046-3736-42b7-9caa-2f0214187790"). InnerVolumeSpecName "kube-api-access-xf9hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:30.938131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.938103 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7864046-3736-42b7-9caa-2f0214187790-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.938131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.938130 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7864046-3736-42b7-9caa-2f0214187790-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.938377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.938141 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9hw\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-kube-api-access-xf9hw\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:30.938377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:30.938151 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7864046-3736-42b7-9caa-2f0214187790-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:31.065848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:31.065814 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn"] Oct 30 19:44:31.667404 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:31.667373 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" event={"ID":"c7864046-3736-42b7-9caa-2f0214187790","Type":"ContainerDied","Data":"5ea6b95388be69a5b9ccb00cc751b39b40c13a3e7a8d3499beaccc6bb42dbe91"} Oct 30 19:44:31.667891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:31.667408 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq" Oct 30 19:44:31.667891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:31.667466 6511 scope.go:122] "RemoveContainer" containerID="7d53d204c90a4201870c78121c3df5969eb929a733d45f43633fceab05c9ee32" Oct 30 19:44:31.709161 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:31.709136 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq"] Oct 30 19:44:31.729296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:31.729274 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1e13df-openshift-default-58bb9b8669-cf9fq"] Oct 30 19:44:32.097306 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:32.097273 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7864046-3736-42b7-9caa-2f0214187790" path="/var/lib/kubelet/pods/c7864046-3736-42b7-9caa-2f0214187790/volumes" Oct 30 19:44:46.086713 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.086654 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" containerID="cri-o://0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e" gracePeriod=45 Oct 30 19:44:46.356666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.356638 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:44:46.461681 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.461649 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c\" (UniqueName: \"kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c\") pod \"c30068df-e7f9-401d-8da8-cd343a8581f4\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " Oct 30 19:44:46.461681 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.461688 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm\") pod \"c30068df-e7f9-401d-8da8-cd343a8581f4\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " Oct 30 19:44:46.461940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.461715 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs\") pod \"c30068df-e7f9-401d-8da8-cd343a8581f4\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " Oct 30 19:44:46.461940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.461750 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home\") pod \"c30068df-e7f9-401d-8da8-cd343a8581f4\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " Oct 30 19:44:46.462058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.461971 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir\") pod \"c30068df-e7f9-401d-8da8-cd343a8581f4\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " Oct 30 19:44:46.462058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462048 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache\") pod \"c30068df-e7f9-401d-8da8-cd343a8581f4\" (UID: \"c30068df-e7f9-401d-8da8-cd343a8581f4\") " Oct 30 19:44:46.462259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462229 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home" pod "c30068df-e7f9-401d-8da8-cd343a8581f4" (UID: "c30068df-e7f9-401d-8da8-cd343a8581f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.462325 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462288 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir" pod "c30068df-e7f9-401d-8da8-cd343a8581f4" (UID: "c30068df-e7f9-401d-8da8-cd343a8581f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.462482 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462462 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache" pod "c30068df-e7f9-401d-8da8-cd343a8581f4" (UID: "c30068df-e7f9-401d-8da8-cd343a8581f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.462634 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462618 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.462701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462635 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.462701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.462646 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.464717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.464692 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs" pod "c30068df-e7f9-401d-8da8-cd343a8581f4" (UID: "c30068df-e7f9-401d-8da8-cd343a8581f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:46.464835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.464810 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c" pod "c30068df-e7f9-401d-8da8-cd343a8581f4" (UID: "c30068df-e7f9-401d-8da8-cd343a8581f4"). InnerVolumeSpecName "kube-api-access-n796c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:46.464835 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.464776 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm" pod "c30068df-e7f9-401d-8da8-cd343a8581f4" (UID: "c30068df-e7f9-401d-8da8-cd343a8581f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:46.563671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.563636 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n796c\" (UniqueName: \"kubernetes.io/projected/c30068df-e7f9-401d-8da8-cd343a8581f4-kube-api-access-n796c\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.563671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.563672 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30068df-e7f9-401d-8da8-cd343a8581f4-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.563865 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.563687 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30068df-e7f9-401d-8da8-cd343a8581f4-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.760168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.760085 6511 generic.go:356] "Generic (PLEG): container finished" podID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerID="0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e" exitCode=0 Oct 30 19:44:46.760168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.760135 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" Oct 30 19:44:46.760493 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.760174 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" event={"ID":"c30068df-e7f9-401d-8da8-cd343a8581f4","Type":"ContainerDied","Data":"0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e"} Oct 30 19:44:46.760493 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.760219 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn" event={"ID":"c30068df-e7f9-401d-8da8-cd343a8581f4","Type":"ContainerDied","Data":"4539c9bdfcfb4ce06a227cfa640a3e6cf49ac76271fbb03c59d0ca83f55517fb"} Oct 30 19:44:46.760493 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.760241 6511 scope.go:122] "RemoveContainer" containerID="0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e" Oct 30 19:44:46.770678 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.770658 6511 scope.go:122] "RemoveContainer" containerID="0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e" Oct 30 19:44:46.770999 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:44:46.770969 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e\": container with ID starting with 0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e not found: ID does not exist" containerID="0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e" Oct 30 19:44:46.771093 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.771008 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e"} err="failed to get container status \"0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e\": rpc error: code = NotFound desc = could not find container \"0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e\": container with ID starting with 0abc1daf938b6a1d58066b8d455dc1ee0327aea48e7f9fe601482b227416176e not found: ID does not exist" Oct 30 19:44:46.796466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.796442 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn"] Oct 30 19:44:46.798713 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:46.798693 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-01b24e/distinct-w6-01b24e-kserve-64fbfd9c75-mvttn"] Oct 30 19:44:48.105571 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:48.105204 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" path="/var/lib/kubelet/pods/c30068df-e7f9-401d-8da8-cd343a8581f4/volumes" Oct 30 19:44:48.106006 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:48.105610 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml"] Oct 30 19:44:48.106006 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:48.105801 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" containerID="cri-o://f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4" gracePeriod=30 Oct 30 19:44:53.372085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.372051 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:44:53.461611 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461508 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.461791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461617 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.461791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461678 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.461791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461722 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.461791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461772 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.462164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461832 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.462164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461859 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.462164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.461931 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.462164 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.462001 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket\") pod \"45d7b3fb-886d-47fa-be1f-b6257175da60\" (UID: \"45d7b3fb-886d-47fa-be1f-b6257175da60\") " Oct 30 19:44:53.462642 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.462611 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:53.463051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.463019 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:53.463296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.463271 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:53.464266 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.464231 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:53.464530 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.464386 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:53.467271 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.467230 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "kube-api-access-cmfh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:53.472254 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.472223 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:53.473691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.473661 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:44:53.474034 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.474007 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token" pod "45d7b3fb-886d-47fa-be1f-b6257175da60" (UID: "45d7b3fb-886d-47fa-be1f-b6257175da60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:53.563933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563898 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.563933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563928 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.563933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563937 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.564147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563948 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d7b3fb-886d-47fa-be1f-b6257175da60-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.564147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563962 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.564147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563974 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.564147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563983 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmfh5\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-kube-api-access-cmfh5\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.564147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.563992 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d7b3fb-886d-47fa-be1f-b6257175da60-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.564147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.564000 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d7b3fb-886d-47fa-be1f-b6257175da60-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:53.811940 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.811903 6511 generic.go:356] "Generic (PLEG): container finished" podID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerID="f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4" exitCode=0 Oct 30 19:44:53.812104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.811958 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" Oct 30 19:44:53.812104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.811992 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" event={"ID":"45d7b3fb-886d-47fa-be1f-b6257175da60","Type":"ContainerDied","Data":"f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4"} Oct 30 19:44:53.812104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.812060 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml" event={"ID":"45d7b3fb-886d-47fa-be1f-b6257175da60","Type":"ContainerDied","Data":"9afee64380276442d9209b708a9377c56d74f78becde815198b3eb19e0d4d784"} Oct 30 19:44:53.812104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.812083 6511 scope.go:122] "RemoveContainer" containerID="f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4" Oct 30 19:44:53.822803 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.822787 6511 scope.go:122] "RemoveContainer" containerID="f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4" Oct 30 19:44:53.823088 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:44:53.823071 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4\": container with ID starting with f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4 not found: ID does not exist" containerID="f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4" Oct 30 19:44:53.823161 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.823097 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4"} err="failed to get container status \"f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4\": rpc error: code = NotFound desc = could not find container \"f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4\": container with ID starting with f1463f9157295db9f6da0360812c00e164684511beccde043754e8c4bbad58a4 not found: ID does not exist" Oct 30 19:44:53.861246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.845107 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml"] Oct 30 19:44:53.887437 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:53.864263 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ac37c8-openshift-default-7c4d5bbb8c-xmzml"] Oct 30 19:44:54.096308 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:54.096235 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" path="/var/lib/kubelet/pods/45d7b3fb-886d-47fa-be1f-b6257175da60/volumes" Oct 30 19:44:54.377953 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:44:54.377871 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:44:59.314655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:59.314614 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6"] Oct 30 19:44:59.327214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:44:59.327185 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94"] Oct 30 19:45:00.202090 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.201972 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp"] Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202640 6511 cpu_manager.go:414] "containerMap: removing container" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202664 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202685 6511 cpu_manager.go:414] "containerMap: removing container" podUID="bdd6751c-1408-4557-82bc-d9e581864563" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202694 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd6751c-1408-4557-82bc-d9e581864563" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202719 6511 cpu_manager.go:414] "containerMap: removing container" podUID="606e46d4-427c-4d9b-b63f-efecaa8d0849" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202726 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606e46d4-427c-4d9b-b63f-efecaa8d0849" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202746 6511 cpu_manager.go:414] "containerMap: removing container" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202753 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202763 6511 cpu_manager.go:414] "containerMap: removing container" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202770 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202795 6511 cpu_manager.go:414] "containerMap: removing container" podUID="4897e42d-9776-4111-9df9-73622a74f67c" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202801 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4897e42d-9776-4111-9df9-73622a74f67c" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202823 6511 cpu_manager.go:414] "containerMap: removing container" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202831 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202841 6511 cpu_manager.go:414] "containerMap: removing container" podUID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" containerName="discovery" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202849 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" containerName="discovery" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202863 6511 cpu_manager.go:414] "containerMap: removing container" podUID="aa7c687e-86c8-4047-844e-ca762470c1ba" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202870 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7c687e-86c8-4047-844e-ca762470c1ba" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202885 6511 cpu_manager.go:414] "containerMap: removing container" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.202894 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203015 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4897e42d-9776-4111-9df9-73622a74f67c" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203031 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abf50bf-3416-46b3-b084-796963443d56" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203039 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c25a6ef-b4b2-4e36-b3cc-847220439ec7" containerName="discovery" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203049 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7c687e-86c8-4047-844e-ca762470c1ba" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203057 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20f0cec-20fa-4300-8838-18f0ef03e9dc" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203067 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d7b3fb-886d-47fa-be1f-b6257175da60" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203078 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd6751c-1408-4557-82bc-d9e581864563" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203088 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="606e46d4-427c-4d9b-b63f-efecaa8d0849" containerName="maas-api" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203099 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7864046-3736-42b7-9caa-2f0214187790" containerName="istio-proxy" Oct 30 19:45:00.203588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.203109 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30068df-e7f9-401d-8da8-cd343a8581f4" containerName="main" Oct 30 19:45:00.206820 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.206801 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.227130 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.227095 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp"] Oct 30 19:45:00.295612 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.295584 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:45:00.364189 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.364143 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7288f\" (UniqueName: \"kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.364584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.364197 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.465568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.465485 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7288f\" (UniqueName: \"kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.465568 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.465539 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.466205 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.466183 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.476519 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.476489 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7288f\" (UniqueName: \"kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.527615 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.527579 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:00.814281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.814250 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp"] Oct 30 19:45:00.815748 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:45:00.815717 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972c6c02_f074_4a31_8cf9_aa5029cc68b4.slice/crio-e8b4bdc858821700da40b62e42b208e32f2d113fa7035a7b2eef2ff2c31975e4 WatchSource:0}: Error finding container e8b4bdc858821700da40b62e42b208e32f2d113fa7035a7b2eef2ff2c31975e4: Status 404 returned error can't find the container with id e8b4bdc858821700da40b62e42b208e32f2d113fa7035a7b2eef2ff2c31975e4 Oct 30 19:45:00.878029 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:00.877983 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" event={"ID":"972c6c02-f074-4a31-8cf9-aa5029cc68b4","Type":"ContainerStarted","Data":"e8b4bdc858821700da40b62e42b208e32f2d113fa7035a7b2eef2ff2c31975e4"} Oct 30 19:45:01.886793 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:01.886754 6511 generic.go:356] "Generic (PLEG): container finished" podID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerID="14566702a4b26fec6a70c1777c048f340ffe970578d6d05a91440a06f8453dcd" exitCode=60 Oct 30 19:45:01.887207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:01.886815 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" event={"ID":"972c6c02-f074-4a31-8cf9-aa5029cc68b4","Type":"ContainerDied","Data":"14566702a4b26fec6a70c1777c048f340ffe970578d6d05a91440a06f8453dcd"} Oct 30 19:45:01.887207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:01.887123 6511 scope.go:122] "RemoveContainer" containerID="14566702a4b26fec6a70c1777c048f340ffe970578d6d05a91440a06f8453dcd" Oct 30 19:45:02.899224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:02.899187 6511 generic.go:356] "Generic (PLEG): container finished" podID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerID="487f5e7b15550792f49077f88b8b0297374107dbe31e24aaf92eb1984ccbd4cd" exitCode=60 Oct 30 19:45:02.899741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:02.899281 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" event={"ID":"972c6c02-f074-4a31-8cf9-aa5029cc68b4","Type":"ContainerDied","Data":"487f5e7b15550792f49077f88b8b0297374107dbe31e24aaf92eb1984ccbd4cd"} Oct 30 19:45:02.899741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:02.899373 6511 scope.go:122] "RemoveContainer" containerID="14566702a4b26fec6a70c1777c048f340ffe970578d6d05a91440a06f8453dcd" Oct 30 19:45:02.899741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:02.899685 6511 scope.go:122] "RemoveContainer" containerID="487f5e7b15550792f49077f88b8b0297374107dbe31e24aaf92eb1984ccbd4cd" Oct 30 19:45:02.900159 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:02.900009 6511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp_odh-ai-gateway-infra(972c6c02-f074-4a31-8cf9-aa5029cc68b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" Oct 30 19:45:03.914483 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:03.914447 6511 scope.go:122] "RemoveContainer" containerID="487f5e7b15550792f49077f88b8b0297374107dbe31e24aaf92eb1984ccbd4cd" Oct 30 19:45:03.914986 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:03.914798 6511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp_odh-ai-gateway-infra(972c6c02-f074-4a31-8cf9-aa5029cc68b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" Oct 30 19:45:09.022466 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:09.022428 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm"] Oct 30 19:45:09.022992 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:09.022945 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" containerID="cri-o://9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5" gracePeriod=30 Oct 30 19:45:09.025095 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:09.025068 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk"] Oct 30 19:45:09.025491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:09.025443 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" containerID="cri-o://6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e" gracePeriod=30 Oct 30 19:45:12.756330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:12.756267 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869"] Oct 30 19:45:12.756870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:12.756576 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" podUID="47ec0193-2f94-4575-b13f-a690ab6dc06e" containerName="maas-api" containerID="cri-o://3fc39258bc515e1effe164fbcce7b7ec9230222f2ddcb0b53f7c91d45be06aaf" gracePeriod=30 Oct 30 19:45:12.977066 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:12.977022 6511 generic.go:356] "Generic (PLEG): container finished" podID="47ec0193-2f94-4575-b13f-a690ab6dc06e" containerID="3fc39258bc515e1effe164fbcce7b7ec9230222f2ddcb0b53f7c91d45be06aaf" exitCode=0 Oct 30 19:45:12.977375 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:12.977344 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" event={"ID":"47ec0193-2f94-4575-b13f-a690ab6dc06e","Type":"ContainerDied","Data":"3fc39258bc515e1effe164fbcce7b7ec9230222f2ddcb0b53f7c91d45be06aaf"} Oct 30 19:45:13.079454 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.072031 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:45:13.155947 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.155917 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw\" (UniqueName: \"kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw\") pod \"47ec0193-2f94-4575-b13f-a690ab6dc06e\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " Oct 30 19:45:13.156132 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.156115 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca\") pod \"47ec0193-2f94-4575-b13f-a690ab6dc06e\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " Oct 30 19:45:13.156220 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.156171 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs\") pod \"47ec0193-2f94-4575-b13f-a690ab6dc06e\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " Oct 30 19:45:13.156336 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.156218 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls\") pod \"47ec0193-2f94-4575-b13f-a690ab6dc06e\" (UID: \"47ec0193-2f94-4575-b13f-a690ab6dc06e\") " Oct 30 19:45:13.156550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.156521 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca" pod "47ec0193-2f94-4575-b13f-a690ab6dc06e" (UID: "47ec0193-2f94-4575-b13f-a690ab6dc06e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:13.156779 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.156752 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ec0193-2f94-4575-b13f-a690ab6dc06e-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:13.158382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.158354 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw" pod "47ec0193-2f94-4575-b13f-a690ab6dc06e" (UID: "47ec0193-2f94-4575-b13f-a690ab6dc06e"). InnerVolumeSpecName "kube-api-access-b2vfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:13.158382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.158353 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs" pod "47ec0193-2f94-4575-b13f-a690ab6dc06e" (UID: "47ec0193-2f94-4575-b13f-a690ab6dc06e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:13.158584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.158496 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls" pod "47ec0193-2f94-4575-b13f-a690ab6dc06e" (UID: "47ec0193-2f94-4575-b13f-a690ab6dc06e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:13.258103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.258070 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:13.258103 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.258103 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vfw\" (UniqueName: \"kubernetes.io/projected/47ec0193-2f94-4575-b13f-a690ab6dc06e-kube-api-access-b2vfw\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:13.258295 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.258114 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ec0193-2f94-4575-b13f-a690ab6dc06e-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:13.985691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.985660 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" Oct 30 19:45:13.985691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.985676 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869" event={"ID":"47ec0193-2f94-4575-b13f-a690ab6dc06e","Type":"ContainerDied","Data":"062e719aa1940435f6af090f841ac1ed3e77d8ec38e9d5e1028cdd1a0713a1ed"} Oct 30 19:45:13.986246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:13.985742 6511 scope.go:122] "RemoveContainer" containerID="3fc39258bc515e1effe164fbcce7b7ec9230222f2ddcb0b53f7c91d45be06aaf" Oct 30 19:45:14.020245 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.020215 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869"] Oct 30 19:45:14.030583 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.030559 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e35df-85b957b874-gr869"] Oct 30 19:45:14.097321 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.097292 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ec0193-2f94-4575-b13f-a690ab6dc06e" path="/var/lib/kubelet/pods/47ec0193-2f94-4575-b13f-a690ab6dc06e/volumes" Oct 30 19:45:14.289981 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.289957 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:45:14.332426 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.332385 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:45:14.335875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.335844 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" containerID="cri-o://8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb" gracePeriod=45 Oct 30 19:45:14.351678 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.351649 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" containerID="cri-o://3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1" gracePeriod=45 Oct 30 19:45:14.374004 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.373967 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.374122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374031 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.374122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374079 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.374197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374124 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.374197 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374167 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.374264 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374249 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.374325 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374308 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374502 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374547 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs\") pod \"f5b0be6f-1786-4ef6-890b-217f626c17f2\" (UID: \"f5b0be6f-1786-4ef6-890b-217f626c17f2\") " Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374560 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.374987 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.375743 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.375819 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.376017 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.377452 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.376080 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.377993 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.377917 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "kube-api-access-dd5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.378165 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.378139 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.378631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.378613 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.379875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.379850 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo" pod "f5b0be6f-1786-4ef6-890b-217f626c17f2" (UID: "f5b0be6f-1786-4ef6-890b-217f626c17f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:14.408976 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:14.408934 6511 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6ad163f_a529_4148_8a0c_da13da341853.slice/crio-conmon-3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:14.475488 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475331 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475488 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475403 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475488 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475467 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475505 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475556 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475704 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475665 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475778 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475816 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.475974 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475878 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data\") pod \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\" (UID: \"ff2ff7d8-7b6b-402e-b14d-692343dd1f22\") " Oct 30 19:45:14.476029 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.475995 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.476300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476265 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.476377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476300 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:14.476377 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476361 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476381 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476397 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476448 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476465 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b0be6f-1786-4ef6-890b-217f626c17f2-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476478 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476493 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd5ql\" (UniqueName: \"kubernetes.io/projected/f5b0be6f-1786-4ef6-890b-217f626c17f2-kube-api-access-dd5ql\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476507 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b0be6f-1786-4ef6-890b-217f626c17f2-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476520 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.476542 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476533 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.477042 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.476645 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.477042 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.477013 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.477994 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.477959 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:14.480278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.478688 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.480278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.479401 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "kube-api-access-52mqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.480278 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.479974 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token" pod "ff2ff7d8-7b6b-402e-b14d-692343dd1f22" (UID: "ff2ff7d8-7b6b-402e-b14d-692343dd1f22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.577403 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577367 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.577562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577406 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.577562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577440 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.577562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577454 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.577562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577469 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.577562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577483 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52mqj\" (UniqueName: \"kubernetes.io/projected/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-kube-api-access-52mqj\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.577562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.577499 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2ff7d8-7b6b-402e-b14d-692343dd1f22-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.602778 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.602099 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:45:14.648797 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.648773 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:45:14.678424 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678387 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home\") pod \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " Oct 30 19:45:14.678569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678489 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir\") pod \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " Oct 30 19:45:14.678569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678526 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22\" (UniqueName: \"kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22\") pod \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " Oct 30 19:45:14.678569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678554 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm\") pod \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " Oct 30 19:45:14.678736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678588 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache\") pod \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " Oct 30 19:45:14.678736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678610 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home" pod "4c9fc379-4173-49c9-aaeb-e85ce3755b0a" (UID: "4c9fc379-4173-49c9-aaeb-e85ce3755b0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.678736 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678695 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs\") pod \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\" (UID: \"4c9fc379-4173-49c9-aaeb-e85ce3755b0a\") " Oct 30 19:45:14.678888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678788 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir" pod "4c9fc379-4173-49c9-aaeb-e85ce3755b0a" (UID: "4c9fc379-4173-49c9-aaeb-e85ce3755b0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.678927 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.678909 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache" pod "4c9fc379-4173-49c9-aaeb-e85ce3755b0a" (UID: "4c9fc379-4173-49c9-aaeb-e85ce3755b0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.679131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.679111 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.679131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.679131 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.679286 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.679145 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.680824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.680798 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm" pod "4c9fc379-4173-49c9-aaeb-e85ce3755b0a" (UID: "4c9fc379-4173-49c9-aaeb-e85ce3755b0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.680902 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.680835 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22" pod "4c9fc379-4173-49c9-aaeb-e85ce3755b0a" (UID: "4c9fc379-4173-49c9-aaeb-e85ce3755b0a"). InnerVolumeSpecName "kube-api-access-hlg22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.680902 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.680851 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs" pod "4c9fc379-4173-49c9-aaeb-e85ce3755b0a" (UID: "4c9fc379-4173-49c9-aaeb-e85ce3755b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:14.779966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.779873 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm\") pod \"c6ad163f-a529-4148-8a0c-da13da341853\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " Oct 30 19:45:14.779966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.779939 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs\") pod \"c6ad163f-a529-4148-8a0c-da13da341853\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " Oct 30 19:45:14.780196 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780003 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x\" (UniqueName: \"kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x\") pod \"c6ad163f-a529-4148-8a0c-da13da341853\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " Oct 30 19:45:14.780196 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780048 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir\") pod \"c6ad163f-a529-4148-8a0c-da13da341853\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " Oct 30 19:45:14.780381 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780361 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir" pod "c6ad163f-a529-4148-8a0c-da13da341853" (UID: "c6ad163f-a529-4148-8a0c-da13da341853"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.780671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780376 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home\") pod \"c6ad163f-a529-4148-8a0c-da13da341853\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " Oct 30 19:45:14.780671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780490 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache\") pod \"c6ad163f-a529-4148-8a0c-da13da341853\" (UID: \"c6ad163f-a529-4148-8a0c-da13da341853\") " Oct 30 19:45:14.780671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780576 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home" pod "c6ad163f-a529-4148-8a0c-da13da341853" (UID: "c6ad163f-a529-4148-8a0c-da13da341853"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.780853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780809 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache" pod "c6ad163f-a529-4148-8a0c-da13da341853" (UID: "c6ad163f-a529-4148-8a0c-da13da341853"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.780992 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.780973 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.781076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.781002 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.781076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.781017 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.781076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.781033 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlg22\" (UniqueName: \"kubernetes.io/projected/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-kube-api-access-hlg22\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.781076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.781049 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.781076 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.781065 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9fc379-4173-49c9-aaeb-e85ce3755b0a-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.782695 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.782663 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm" pod "c6ad163f-a529-4148-8a0c-da13da341853" (UID: "c6ad163f-a529-4148-8a0c-da13da341853"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:14.782812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.782756 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs" pod "c6ad163f-a529-4148-8a0c-da13da341853" (UID: "c6ad163f-a529-4148-8a0c-da13da341853"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:14.782812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.782806 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x" pod "c6ad163f-a529-4148-8a0c-da13da341853" (UID: "c6ad163f-a529-4148-8a0c-da13da341853"). InnerVolumeSpecName "kube-api-access-dwq4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:14.881997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.881965 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ad163f-a529-4148-8a0c-da13da341853-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.881997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.881994 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwq4x\" (UniqueName: \"kubernetes.io/projected/c6ad163f-a529-4148-8a0c-da13da341853-kube-api-access-dwq4x\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.881997 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.882004 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ad163f-a529-4148-8a0c-da13da341853-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:14.993398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.993363 6511 generic.go:356] "Generic (PLEG): container finished" podID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerID="6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e" exitCode=0 Oct 30 19:45:14.993833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.993434 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" Oct 30 19:45:14.993833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.993458 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" event={"ID":"ff2ff7d8-7b6b-402e-b14d-692343dd1f22","Type":"ContainerDied","Data":"6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e"} Oct 30 19:45:14.993833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.993513 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk" event={"ID":"ff2ff7d8-7b6b-402e-b14d-692343dd1f22","Type":"ContainerDied","Data":"daa1bca035fdb0e79b17d2834427be983c00819d885ebb84c17b6707ef8e658c"} Oct 30 19:45:14.993833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.993542 6511 scope.go:122] "RemoveContainer" containerID="6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e" Oct 30 19:45:14.995453 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.995400 6511 generic.go:356] "Generic (PLEG): container finished" podID="c6ad163f-a529-4148-8a0c-da13da341853" containerID="3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1" exitCode=0 Oct 30 19:45:14.995560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.995542 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" event={"ID":"c6ad163f-a529-4148-8a0c-da13da341853","Type":"ContainerDied","Data":"3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1"} Oct 30 19:45:14.995612 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.995577 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" event={"ID":"c6ad163f-a529-4148-8a0c-da13da341853","Type":"ContainerDied","Data":"1984d6245215403a2ed5d6fd3a925505bd830a0efda65920525a01bf1ca41101"} Oct 30 19:45:14.995661 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:14.995635 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94" Oct 30 19:45:15.000105 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.000082 6511 generic.go:356] "Generic (PLEG): container finished" podID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerID="8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb" exitCode=0 Oct 30 19:45:15.000216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.000133 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" Oct 30 19:45:15.000216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.000174 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" event={"ID":"4c9fc379-4173-49c9-aaeb-e85ce3755b0a","Type":"ContainerDied","Data":"8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb"} Oct 30 19:45:15.000216 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.000205 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6" event={"ID":"4c9fc379-4173-49c9-aaeb-e85ce3755b0a","Type":"ContainerDied","Data":"a6d93da6129f720decfcb4bd7cc83f496f71a55e37d7305c644da3896ea2130e"} Oct 30 19:45:15.001604 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.001584 6511 generic.go:356] "Generic (PLEG): container finished" podID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerID="9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5" exitCode=0 Oct 30 19:45:15.001697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.001627 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" Oct 30 19:45:15.001697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.001628 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" event={"ID":"f5b0be6f-1786-4ef6-890b-217f626c17f2","Type":"ContainerDied","Data":"9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5"} Oct 30 19:45:15.001697 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.001669 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm" event={"ID":"f5b0be6f-1786-4ef6-890b-217f626c17f2","Type":"ContainerDied","Data":"07efa832883f84169eec18ee3c763f7dd8ff5809da40394cd8196c19ad8dda5e"} Oct 30 19:45:15.005765 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.005749 6511 scope.go:122] "RemoveContainer" containerID="6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e" Oct 30 19:45:15.006470 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:15.006051 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e\": container with ID starting with 6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e not found: ID does not exist" containerID="6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e" Oct 30 19:45:15.006470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.006090 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e"} err="failed to get container status \"6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e\": rpc error: code = NotFound desc = could not find container \"6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e\": container with ID starting with 6b4f8aa71192ac7c8ad826656ca525a32eb9d7c23120c59b9d2dce564f63fe5e not found: ID does not exist" Oct 30 19:45:15.006470 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.006112 6511 scope.go:122] "RemoveContainer" containerID="3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1" Oct 30 19:45:15.017374 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.017347 6511 scope.go:122] "RemoveContainer" containerID="3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1" Oct 30 19:45:15.017705 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:15.017677 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1\": container with ID starting with 3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1 not found: ID does not exist" containerID="3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1" Oct 30 19:45:15.017814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.017710 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1"} err="failed to get container status \"3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1\": rpc error: code = NotFound desc = could not find container \"3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1\": container with ID starting with 3d4717ce9fc5a7dc4a70786a20f78a5c4dcb945b6ed0966c5d06c19a095ce4c1 not found: ID does not exist" Oct 30 19:45:15.017814 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.017731 6511 scope.go:122] "RemoveContainer" containerID="8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb" Oct 30 19:45:15.039819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.036288 6511 scope.go:122] "RemoveContainer" containerID="8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb" Oct 30 19:45:15.039819 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:15.036898 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb\": container with ID starting with 8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb not found: ID does not exist" containerID="8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb" Oct 30 19:45:15.039819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.036940 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb"} err="failed to get container status \"8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb\": rpc error: code = NotFound desc = could not find container \"8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb\": container with ID starting with 8643427a012d5406435e0b024148c1efa3f547b8b5ce3d65b4c705f61399cfeb not found: ID does not exist" Oct 30 19:45:15.039819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.036974 6511 scope.go:122] "RemoveContainer" containerID="9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5" Oct 30 19:45:15.054359 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.054313 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94"] Oct 30 19:45:15.056663 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.056642 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-w1-419d0a-kserve-cd5dc97cc-jrp94"] Oct 30 19:45:15.072875 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.072853 6511 scope.go:122] "RemoveContainer" containerID="9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5" Oct 30 19:45:15.073232 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:15.073209 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5\": container with ID starting with 9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5 not found: ID does not exist" containerID="9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5" Oct 30 19:45:15.073334 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.073241 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5"} err="failed to get container status \"9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5\": rpc error: code = NotFound desc = could not find container \"9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5\": container with ID starting with 9c5c26e07aa9c04f402ddb37a453fc5c3a4ac1ba5728ee6a8177e302369818b5 not found: ID does not exist" Oct 30 19:45:15.080799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.080772 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk"] Oct 30 19:45:15.092045 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.092018 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-t8wnk"] Oct 30 19:45:15.123642 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.123610 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm"] Oct 30 19:45:15.134056 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.134018 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-01b24e-openshift-default-6f9bfc6d77-vz4wm"] Oct 30 19:45:15.165721 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.165681 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6"] Oct 30 19:45:15.185723 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:15.185696 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-419d0a/distinct-2-w1-419d0a-kserve-5cdd8b8786-wq2k6"] Oct 30 19:45:16.098389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:16.098360 6511 scope.go:122] "RemoveContainer" containerID="487f5e7b15550792f49077f88b8b0297374107dbe31e24aaf92eb1984ccbd4cd" Oct 30 19:45:16.105750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:16.105720 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" path="/var/lib/kubelet/pods/4c9fc379-4173-49c9-aaeb-e85ce3755b0a/volumes" Oct 30 19:45:16.106431 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:16.106390 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ad163f-a529-4148-8a0c-da13da341853" path="/var/lib/kubelet/pods/c6ad163f-a529-4148-8a0c-da13da341853/volumes" Oct 30 19:45:16.107226 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:16.107046 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" path="/var/lib/kubelet/pods/f5b0be6f-1786-4ef6-890b-217f626c17f2/volumes" Oct 30 19:45:16.108098 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:16.108077 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" path="/var/lib/kubelet/pods/ff2ff7d8-7b6b-402e-b14d-692343dd1f22/volumes" Oct 30 19:45:17.020966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:17.020935 6511 generic.go:356] "Generic (PLEG): container finished" podID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerID="365367a29763619e4a5eb58972abf335d3d70e041d4023e99d3d56fa018ede5e" exitCode=60 Oct 30 19:45:17.021104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:17.020995 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" event={"ID":"972c6c02-f074-4a31-8cf9-aa5029cc68b4","Type":"ContainerDied","Data":"365367a29763619e4a5eb58972abf335d3d70e041d4023e99d3d56fa018ede5e"} Oct 30 19:45:17.021104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:17.021052 6511 scope.go:122] "RemoveContainer" containerID="487f5e7b15550792f49077f88b8b0297374107dbe31e24aaf92eb1984ccbd4cd" Oct 30 19:45:17.021439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:17.021389 6511 scope.go:122] "RemoveContainer" containerID="365367a29763619e4a5eb58972abf335d3d70e041d4023e99d3d56fa018ede5e" Oct 30 19:45:17.021697 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:17.021675 6511 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp_odh-ai-gateway-infra(972c6c02-f074-4a31-8cf9-aa5029cc68b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" Oct 30 19:45:18.096907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.096861 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp"] Oct 30 19:45:18.237786 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.237758 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:18.323959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.320757 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca\") pod \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " Oct 30 19:45:18.323959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.320876 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f\" (UniqueName: \"kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f\") pod \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\" (UID: \"972c6c02-f074-4a31-8cf9-aa5029cc68b4\") " Oct 30 19:45:18.323959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.322119 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca" pod "972c6c02-f074-4a31-8cf9-aa5029cc68b4" (UID: "972c6c02-f074-4a31-8cf9-aa5029cc68b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.329085 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.329053 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f" pod "972c6c02-f074-4a31-8cf9-aa5029cc68b4" (UID: "972c6c02-f074-4a31-8cf9-aa5029cc68b4"). InnerVolumeSpecName "kube-api-access-7288f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.422347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.422272 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/972c6c02-f074-4a31-8cf9-aa5029cc68b4-maas-api-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.422347 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:18.422299 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7288f\" (UniqueName: \"kubernetes.io/projected/972c6c02-f074-4a31-8cf9-aa5029cc68b4-kube-api-access-7288f\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.036688 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:19.036657 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" Oct 30 19:45:19.036894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:19.036705 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp" event={"ID":"972c6c02-f074-4a31-8cf9-aa5029cc68b4","Type":"ContainerDied","Data":"e8b4bdc858821700da40b62e42b208e32f2d113fa7035a7b2eef2ff2c31975e4"} Oct 30 19:45:19.036894 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:19.036777 6511 scope.go:122] "RemoveContainer" containerID="365367a29763619e4a5eb58972abf335d3d70e041d4023e99d3d56fa018ede5e" Oct 30 19:45:19.067921 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:19.067894 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp"] Oct 30 19:45:19.080224 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:19.080202 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-976b80-29889825-gskjp"] Oct 30 19:45:20.095820 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:20.095780 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" path="/var/lib/kubelet/pods/972c6c02-f074-4a31-8cf9-aa5029cc68b4/volumes" Oct 30 19:45:24.377789 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:24.377741 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:45:24.464343 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.464311 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp"] Oct 30 19:45:24.464576 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.464549 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" podUID="c73793a2-8c80-4122-b5d9-36a628bec9f1" containerName="maas-api" containerID="cri-o://01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841" gracePeriod=30 Oct 30 19:45:24.583657 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.583621 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn"] Oct 30 19:45:24.583915 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.583886 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" containerID="cri-o://e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c" gracePeriod=30 Oct 30 19:45:24.599720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.599681 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" podUID="c73793a2-8c80-4122-b5d9-36a628bec9f1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.38:8443/health\": dial tcp 10.133.0.38:8443: connect: connection refused" Oct 30 19:45:24.743170 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.743145 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:45:24.895474 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.895445 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls\") pod \"c73793a2-8c80-4122-b5d9-36a628bec9f1\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " Oct 30 19:45:24.895652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.895534 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z\" (UniqueName: \"kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z\") pod \"c73793a2-8c80-4122-b5d9-36a628bec9f1\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " Oct 30 19:45:24.895652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.895566 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs\") pod \"c73793a2-8c80-4122-b5d9-36a628bec9f1\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " Oct 30 19:45:24.895739 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.895651 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca\") pod \"c73793a2-8c80-4122-b5d9-36a628bec9f1\" (UID: \"c73793a2-8c80-4122-b5d9-36a628bec9f1\") " Oct 30 19:45:24.896055 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.896030 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca" pod "c73793a2-8c80-4122-b5d9-36a628bec9f1" (UID: "c73793a2-8c80-4122-b5d9-36a628bec9f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:24.897861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.897808 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z" pod "c73793a2-8c80-4122-b5d9-36a628bec9f1" (UID: "c73793a2-8c80-4122-b5d9-36a628bec9f1"). InnerVolumeSpecName "kube-api-access-hpp5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:24.898236 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.898218 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs" pod "c73793a2-8c80-4122-b5d9-36a628bec9f1" (UID: "c73793a2-8c80-4122-b5d9-36a628bec9f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:24.898283 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.898246 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls" pod "c73793a2-8c80-4122-b5d9-36a628bec9f1" (UID: "c73793a2-8c80-4122-b5d9-36a628bec9f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:24.996907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.996871 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpp5z\" (UniqueName: \"kubernetes.io/projected/c73793a2-8c80-4122-b5d9-36a628bec9f1-kube-api-access-hpp5z\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:24.996907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.996901 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:24.996907 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.996911 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73793a2-8c80-4122-b5d9-36a628bec9f1-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:24.997134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:24.996921 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c73793a2-8c80-4122-b5d9-36a628bec9f1-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:25.090213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.089687 6511 generic.go:356] "Generic (PLEG): container finished" podID="c73793a2-8c80-4122-b5d9-36a628bec9f1" containerID="01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841" exitCode=0 Oct 30 19:45:25.090213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.089903 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" event={"ID":"c73793a2-8c80-4122-b5d9-36a628bec9f1","Type":"ContainerDied","Data":"01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841"} Oct 30 19:45:25.090213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.089992 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" event={"ID":"c73793a2-8c80-4122-b5d9-36a628bec9f1","Type":"ContainerDied","Data":"cc8f02879510de3ec6e71946b9f6ee6fd8b515dcdf5b62d62844fe4558c36d38"} Oct 30 19:45:25.090213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.090027 6511 scope.go:122] "RemoveContainer" containerID="01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841" Oct 30 19:45:25.090213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.090171 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp" Oct 30 19:45:25.102637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.102613 6511 scope.go:122] "RemoveContainer" containerID="01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841" Oct 30 19:45:25.102954 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:25.102933 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841\": container with ID starting with 01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841 not found: ID does not exist" containerID="01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841" Oct 30 19:45:25.103005 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.102964 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841"} err="failed to get container status \"01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841\": rpc error: code = NotFound desc = could not find container \"01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841\": container with ID starting with 01581cad60bcee5acae97d691e6ef301267d95df6a8a124061b12ece11b49841 not found: ID does not exist" Oct 30 19:45:25.122916 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.122881 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp"] Oct 30 19:45:25.130432 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:25.130392 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-419d0a-7967fdbdb9-k27zp"] Oct 30 19:45:26.100188 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:26.100142 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73793a2-8c80-4122-b5d9-36a628bec9f1" path="/var/lib/kubelet/pods/c73793a2-8c80-4122-b5d9-36a628bec9f1/volumes" Oct 30 19:45:27.582565 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:27.582529 6511 patch_prober.go:29] interesting pod/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:27.582943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:27.582593 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:29.848439 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.848368 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:45:29.946630 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946595 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.946817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946643 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.946817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946678 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.946817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946740 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.946817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946770 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.946817 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946799 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.947116 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946870 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.947116 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946915 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.947116 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.946976 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q\") pod \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\" (UID: \"db0ffd8f-5549-49ed-9d8c-b30cc8474977\") " Oct 30 19:45:29.947285 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947219 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:29.947285 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947270 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:29.947398 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947354 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:29.947977 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947627 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:29.948089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947993 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:29.948089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.948012 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:29.948089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947754 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:29.948089 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.947936 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:29.949207 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.949186 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:29.949569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.949536 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "kube-api-access-d8w5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:29.949655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.949607 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:29.949655 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:29.949621 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token" pod "db0ffd8f-5549-49ed-9d8c-b30cc8474977" (UID: "db0ffd8f-5549-49ed-9d8c-b30cc8474977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:30.048584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.048557 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.048584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.048584 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.048761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.048594 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8w5q\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-kube-api-access-d8w5q\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.048761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.048603 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.048761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.048611 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.048761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.048619 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0ffd8f-5549-49ed-9d8c-b30cc8474977-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:30.124701 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.124673 6511 generic.go:356] "Generic (PLEG): container finished" podID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerID="e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c" exitCode=0 Oct 30 19:45:30.124854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.124727 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" Oct 30 19:45:30.124854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.124761 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" event={"ID":"db0ffd8f-5549-49ed-9d8c-b30cc8474977","Type":"ContainerDied","Data":"e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c"} Oct 30 19:45:30.124854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.124827 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn" event={"ID":"db0ffd8f-5549-49ed-9d8c-b30cc8474977","Type":"ContainerDied","Data":"a3632718965c480950579e5c0ad8aa87aa260709e727199dc6060c8bea9ec9d9"} Oct 30 19:45:30.125078 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.124855 6511 scope.go:122] "RemoveContainer" containerID="e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c" Oct 30 19:45:30.135261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.135236 6511 scope.go:122] "RemoveContainer" containerID="e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f" Oct 30 19:45:30.144012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.143994 6511 scope.go:122] "RemoveContainer" containerID="e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c" Oct 30 19:45:30.144554 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:30.144527 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c\": container with ID starting with e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c not found: ID does not exist" containerID="e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c" Oct 30 19:45:30.144637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.144569 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c"} err="failed to get container status \"e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c\": rpc error: code = NotFound desc = could not find container \"e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c\": container with ID starting with e74bfaec0653adfdaac84ee06c2166222c85bfcceb22e0593f016ee9ddb4e59c not found: ID does not exist" Oct 30 19:45:30.144637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.144586 6511 scope.go:122] "RemoveContainer" containerID="e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f" Oct 30 19:45:30.144821 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:30.144801 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f\": container with ID starting with e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f not found: ID does not exist" containerID="e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f" Oct 30 19:45:30.144897 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.144834 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f"} err="failed to get container status \"e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f\": rpc error: code = NotFound desc = could not find container \"e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f\": container with ID starting with e196cf773de68b38daadfe17bb9de45b10da4943d3dcf7d7ed3f33fdfaaaca9f not found: ID does not exist" Oct 30 19:45:30.148155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.148097 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn"] Oct 30 19:45:30.153372 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:30.153343 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3e35df-openshift-default-67646bf575-7psjn"] Oct 30 19:45:32.097824 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:32.097787 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" path="/var/lib/kubelet/pods/db0ffd8f-5549-49ed-9d8c-b30cc8474977/volumes" Oct 30 19:45:35.293888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:35.293268 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb"] Oct 30 19:45:35.293888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:35.293540 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" containerID="cri-o://7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5" gracePeriod=30 Oct 30 19:45:37.074279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.074241 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb"] Oct 30 19:45:37.074703 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.074552 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" podUID="4a8e2f87-4f63-48c1-af13-b502e7fef285" containerName="maas-api" containerID="cri-o://a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803" gracePeriod=30 Oct 30 19:45:37.346220 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.346191 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:45:37.438059 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.438016 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca\") pod \"4a8e2f87-4f63-48c1-af13-b502e7fef285\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " Oct 30 19:45:37.438228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.438109 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs\") pod \"4a8e2f87-4f63-48c1-af13-b502e7fef285\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " Oct 30 19:45:37.438228 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.438209 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r\" (UniqueName: \"kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r\") pod \"4a8e2f87-4f63-48c1-af13-b502e7fef285\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " Oct 30 19:45:37.438380 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.438259 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls\") pod \"4a8e2f87-4f63-48c1-af13-b502e7fef285\" (UID: \"4a8e2f87-4f63-48c1-af13-b502e7fef285\") " Oct 30 19:45:37.438491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.438465 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca" pod "4a8e2f87-4f63-48c1-af13-b502e7fef285" (UID: "4a8e2f87-4f63-48c1-af13-b502e7fef285"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:37.438934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.438812 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a8e2f87-4f63-48c1-af13-b502e7fef285-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:37.440513 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.440480 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls" pod "4a8e2f87-4f63-48c1-af13-b502e7fef285" (UID: "4a8e2f87-4f63-48c1-af13-b502e7fef285"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:37.440597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.440525 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs" pod "4a8e2f87-4f63-48c1-af13-b502e7fef285" (UID: "4a8e2f87-4f63-48c1-af13-b502e7fef285"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:37.440597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.440561 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r" pod "4a8e2f87-4f63-48c1-af13-b502e7fef285" (UID: "4a8e2f87-4f63-48c1-af13-b502e7fef285"). InnerVolumeSpecName "kube-api-access-xht5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:37.540313 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.540275 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:37.540313 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.540315 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xht5r\" (UniqueName: \"kubernetes.io/projected/4a8e2f87-4f63-48c1-af13-b502e7fef285-kube-api-access-xht5r\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:37.540535 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:37.540329 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a8e2f87-4f63-48c1-af13-b502e7fef285-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:38.172594 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.172562 6511 generic.go:356] "Generic (PLEG): container finished" podID="4a8e2f87-4f63-48c1-af13-b502e7fef285" containerID="a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803" exitCode=0 Oct 30 19:45:38.172991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.172612 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" Oct 30 19:45:38.172991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.172616 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" event={"ID":"4a8e2f87-4f63-48c1-af13-b502e7fef285","Type":"ContainerDied","Data":"a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803"} Oct 30 19:45:38.172991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.172739 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb" event={"ID":"4a8e2f87-4f63-48c1-af13-b502e7fef285","Type":"ContainerDied","Data":"5261f784297babe47d7f3d3bbea01f5baf5c5f985bbbc127f3b685be6e56ff2b"} Oct 30 19:45:38.172991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.172767 6511 scope.go:122] "RemoveContainer" containerID="a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803" Oct 30 19:45:38.182072 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.182054 6511 scope.go:122] "RemoveContainer" containerID="a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803" Oct 30 19:45:38.182364 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:38.182345 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803\": container with ID starting with a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803 not found: ID does not exist" containerID="a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803" Oct 30 19:45:38.182436 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.182373 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803"} err="failed to get container status \"a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803\": rpc error: code = NotFound desc = could not find container \"a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803\": container with ID starting with a9b60e3baba3a539fb1563d06a6ee600f8fce755a192f54e5b3f414fe1463803 not found: ID does not exist" Oct 30 19:45:38.196769 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.196745 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb"] Oct 30 19:45:38.204871 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:38.204850 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-47110f-9945c9b5d-znctb"] Oct 30 19:45:40.096804 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.096768 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8e2f87-4f63-48c1-af13-b502e7fef285" path="/var/lib/kubelet/pods/4a8e2f87-4f63-48c1-af13-b502e7fef285/volumes" Oct 30 19:45:40.577260 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.577232 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:45:40.683213 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683136 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683219 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683286 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683349 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683378 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683441 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683485 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683477 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.683666 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683503 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.683887 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683761 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.684167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.683966 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.684167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.684137 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token\") pod \"30f52211-5af3-426c-8be3-c486c14bd2dc\" (UID: \"30f52211-5af3-426c-8be3-c486c14bd2dc\") " Oct 30 19:45:40.684609 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.684584 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.684705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.684615 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.684705 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.684631 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.684823 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.684799 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:40.684895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.684878 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.685852 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.685827 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:40.686168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.686140 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:40.686306 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.686286 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "kube-api-access-26hcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:40.686562 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.686539 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token" pod "30f52211-5af3-426c-8be3-c486c14bd2dc" (UID: "30f52211-5af3-426c-8be3-c486c14bd2dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:40.785686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.785646 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.785686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.785680 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.785686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.785693 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.785917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.785704 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30f52211-5af3-426c-8be3-c486c14bd2dc-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.785917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.785717 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hcw\" (UniqueName: \"kubernetes.io/projected/30f52211-5af3-426c-8be3-c486c14bd2dc-kube-api-access-26hcw\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:40.785917 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:40.785730 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30f52211-5af3-426c-8be3-c486c14bd2dc-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:41.193895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.193860 6511 generic.go:356] "Generic (PLEG): container finished" podID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerID="7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5" exitCode=0 Oct 30 19:45:41.194309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.193945 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" Oct 30 19:45:41.194309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.193947 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" event={"ID":"30f52211-5af3-426c-8be3-c486c14bd2dc","Type":"ContainerDied","Data":"7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5"} Oct 30 19:45:41.194309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.194029 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb" event={"ID":"30f52211-5af3-426c-8be3-c486c14bd2dc","Type":"ContainerDied","Data":"bf08152dd60b4c05618fea335cf499cf593d4fb61c13f339c3d0555aec60ebb9"} Oct 30 19:45:41.194309 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.194051 6511 scope.go:122] "RemoveContainer" containerID="7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5" Oct 30 19:45:41.204843 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.204825 6511 scope.go:122] "RemoveContainer" containerID="7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5" Oct 30 19:45:41.205147 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:41.205120 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5\": container with ID starting with 7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5 not found: ID does not exist" containerID="7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5" Oct 30 19:45:41.205246 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.205161 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5"} err="failed to get container status \"7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5\": rpc error: code = NotFound desc = could not find container \"7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5\": container with ID starting with 7eda519448f039b58583f30014c68f342d23094c45c3fa55f274203f801d4ea5 not found: ID does not exist" Oct 30 19:45:41.230337 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.230312 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb"] Oct 30 19:45:41.242545 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:41.242522 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-419d0a-openshift-default-55766cf857-fqcxb"] Oct 30 19:45:42.096382 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:42.096351 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" path="/var/lib/kubelet/pods/30f52211-5af3-426c-8be3-c486c14bd2dc/volumes" Oct 30 19:45:43.853955 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:43.853916 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz"] Oct 30 19:45:47.609250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:47.609168 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r"] Oct 30 19:45:47.609717 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:47.609408 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" containerID="cri-o://7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0" gracePeriod=30 Oct 30 19:45:49.374300 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374268 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p"] Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374686 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374699 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374713 6511 cpu_manager.go:414] "containerMap: removing container" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374719 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374730 6511 cpu_manager.go:414] "containerMap: removing container" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374735 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374743 6511 cpu_manager.go:414] "containerMap: removing container" podUID="47ec0193-2f94-4575-b13f-a690ab6dc06e" containerName="maas-api" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374747 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ec0193-2f94-4575-b13f-a690ab6dc06e" containerName="maas-api" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374758 6511 cpu_manager.go:414] "containerMap: removing container" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374762 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374768 6511 cpu_manager.go:414] "containerMap: removing container" podUID="c73793a2-8c80-4122-b5d9-36a628bec9f1" containerName="maas-api" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374772 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73793a2-8c80-4122-b5d9-36a628bec9f1" containerName="maas-api" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374782 6511 cpu_manager.go:414] "containerMap: removing container" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374786 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374797 6511 cpu_manager.go:414] "containerMap: removing container" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374801 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374807 6511 cpu_manager.go:414] "containerMap: removing container" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374811 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374819 6511 cpu_manager.go:414] "containerMap: removing container" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.374816 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374823 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374830 6511 cpu_manager.go:414] "containerMap: removing container" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374852 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374860 6511 cpu_manager.go:414] "containerMap: removing container" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374864 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374871 6511 cpu_manager.go:414] "containerMap: removing container" podUID="4a8e2f87-4f63-48c1-af13-b502e7fef285" containerName="maas-api" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374878 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8e2f87-4f63-48c1-af13-b502e7fef285" containerName="maas-api" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374935 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f52211-5af3-426c-8be3-c486c14bd2dc" containerName="istio-proxy" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374944 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374949 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374953 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2ff7d8-7b6b-402e-b14d-692343dd1f22" containerName="istio-proxy" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374959 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ad163f-a529-4148-8a0c-da13da341853" containerName="main" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374964 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8e2f87-4f63-48c1-af13-b502e7fef285" containerName="maas-api" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374973 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b0be6f-1786-4ef6-890b-217f626c17f2" containerName="istio-proxy" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374981 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9fc379-4173-49c9-aaeb-e85ce3755b0a" containerName="main" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374986 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ec0193-2f94-4575-b13f-a690ab6dc06e" containerName="maas-api" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374991 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:45:49.375533 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.374995 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73793a2-8c80-4122-b5d9-36a628bec9f1" containerName="maas-api" Oct 30 19:45:49.379110 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.379085 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.386009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.385983 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:49.386158 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.386107 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vvmk9\"" Oct 30 19:45:49.386304 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.386281 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p"] Oct 30 19:45:49.477399 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.477363 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk46q\" (UniqueName: \"kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.477399 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.477402 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.477668 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.477563 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.578925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.578883 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk46q\" (UniqueName: \"kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.579119 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.578940 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.579119 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.579008 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.580002 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.579976 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.582323 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.582291 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.588763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.588733 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk46q\" (UniqueName: \"kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q\") pod \"maas-api-revoke-sub-27026cb4-wbx2p\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.696810 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.696713 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:49.995867 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:49.995788 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p"] Oct 30 19:45:49.999675 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:45:49.999643 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb4cfcf_8979_4b82_b7ea_a92f7de2613b.slice/crio-987dfc878edaa2b5e1518481a73d55e5c7014d7873b16c607ccd54faf2672161 WatchSource:0}: Error finding container 987dfc878edaa2b5e1518481a73d55e5c7014d7873b16c607ccd54faf2672161: Status 404 returned error can't find the container with id 987dfc878edaa2b5e1518481a73d55e5c7014d7873b16c607ccd54faf2672161 Oct 30 19:45:50.253297 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:50.253208 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" event={"ID":"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b","Type":"ContainerStarted","Data":"987dfc878edaa2b5e1518481a73d55e5c7014d7873b16c607ccd54faf2672161"} Oct 30 19:45:51.260550 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:51.260508 6511 generic.go:356] "Generic (PLEG): container finished" podID="9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" containerID="18e842cd629393f2160991a977cc411e7b2877b4a569856fd82f1d919a71f3a3" exitCode=0 Oct 30 19:45:51.260983 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:51.260550 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" event={"ID":"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b","Type":"ContainerDied","Data":"18e842cd629393f2160991a977cc411e7b2877b4a569856fd82f1d919a71f3a3"} Oct 30 19:45:51.398699 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:51.398660 6511 patch_prober.go:29] interesting pod/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:51.398873 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:51.398720 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:52.406632 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.406603 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:52.507144 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.507106 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca\") pod \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " Oct 30 19:45:52.507344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.507254 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca\") pod \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " Oct 30 19:45:52.507344 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.507276 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q\" (UniqueName: \"kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q\") pod \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\" (UID: \"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b\") " Oct 30 19:45:52.507499 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.507476 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca" pod "9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" (UID: "9fb4cfcf-8979-4b82-b7ea-a92f7de2613b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:52.507620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.507604 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-openshift-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.509486 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.509453 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca" pod "9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" (UID: "9fb4cfcf-8979-4b82-b7ea-a92f7de2613b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:52.509641 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.509619 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q" pod "9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" (UID: "9fb4cfcf-8979-4b82-b7ea-a92f7de2613b"). InnerVolumeSpecName "kube-api-access-vk46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:52.608801 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.608773 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-maas-api-xks-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.608801 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.608800 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk46q\" (UniqueName: \"kubernetes.io/projected/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b-kube-api-access-vk46q\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:52.865841 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:52.865775 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:45:53.013891 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.013861 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.013912 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.013942 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014053 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.013985 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014060 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014087 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014116 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014214 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014187 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014228 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token\") pod \"1b6c6b51-c969-451b-b1fe-6287758d2679\" (UID: \"1b6c6b51-c969-451b-b1fe-6287758d2679\") " Oct 30 19:45:53.014401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014226 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:53.014401 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014389 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:53.014585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014488 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:53.014585 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014536 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:53.014774 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014754 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.014854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014778 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b6c6b51-c969-451b-b1fe-6287758d2679-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.014854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014792 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.014854 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014807 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.015022 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.014925 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:53.016126 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.016105 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:53.016491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.016473 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:53.016584 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.016522 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:53.016741 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.016725 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x" pod "1b6c6b51-c969-451b-b1fe-6287758d2679" (UID: "1b6c6b51-c969-451b-b1fe-6287758d2679"). InnerVolumeSpecName "kube-api-access-kzj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:53.115389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.115333 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzj6x\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-kube-api-access-kzj6x\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.115389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.115357 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.115389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.115368 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.115389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.115379 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.115389 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.115387 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b6c6b51-c969-451b-b1fe-6287758d2679-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.148123 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.148094 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227"] Oct 30 19:45:53.173461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.173434 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd"] Oct 30 19:45:53.274237 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.274206 6511 generic.go:356] "Generic (PLEG): container finished" podID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerID="7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0" exitCode=0 Oct 30 19:45:53.274457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.274248 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" Oct 30 19:45:53.274457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.274297 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" event={"ID":"1b6c6b51-c969-451b-b1fe-6287758d2679","Type":"ContainerDied","Data":"7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0"} Oct 30 19:45:53.274457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.274367 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r" event={"ID":"1b6c6b51-c969-451b-b1fe-6287758d2679","Type":"ContainerDied","Data":"29722d318471a91e3804139578806f154a9634d6375e1f6ff6ca065a28f00022"} Oct 30 19:45:53.274457 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.274403 6511 scope.go:122] "RemoveContainer" containerID="7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0" Oct 30 19:45:53.275934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.275859 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" event={"ID":"9fb4cfcf-8979-4b82-b7ea-a92f7de2613b","Type":"ContainerDied","Data":"987dfc878edaa2b5e1518481a73d55e5c7014d7873b16c607ccd54faf2672161"} Oct 30 19:45:53.275934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.275900 6511 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="987dfc878edaa2b5e1518481a73d55e5c7014d7873b16c607ccd54faf2672161" Oct 30 19:45:53.275934 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.275926 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p" Oct 30 19:45:53.285135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.285118 6511 scope.go:122] "RemoveContainer" containerID="7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0" Oct 30 19:45:53.285451 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:53.285396 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0\": container with ID starting with 7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0 not found: ID does not exist" containerID="7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0" Oct 30 19:45:53.285554 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.285513 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0"} err="failed to get container status \"7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0\": rpc error: code = NotFound desc = could not find container \"7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0\": container with ID starting with 7bd2dcc721854bd5bf187819614ba4844f4a2d85631114e0e01817a79ec56fe0 not found: ID does not exist" Oct 30 19:45:53.331966 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.331936 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r"] Oct 30 19:45:53.344157 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:53.344130 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-47110f-openshift-default-6dc75dc96c-dcf9r"] Oct 30 19:45:54.102400 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:54.102361 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" path="/var/lib/kubelet/pods/1b6c6b51-c969-451b-b1fe-6287758d2679/volumes" Oct 30 19:45:54.377519 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:54.377442 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:45:58.875841 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:58.875774 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" containerID="cri-o://235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3" gracePeriod=45 Oct 30 19:45:59.130337 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.130280 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:45:59.281158 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281128 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs\") pod \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " Oct 30 19:45:59.281356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281196 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg\" (UniqueName: \"kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg\") pod \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " Oct 30 19:45:59.281356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281234 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home\") pod \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " Oct 30 19:45:59.281356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281259 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache\") pod \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " Oct 30 19:45:59.281356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281296 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir\") pod \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " Oct 30 19:45:59.281356 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281325 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm\") pod \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\" (UID: \"3c64d263-5c5a-4c5b-891c-4607e6e850a3\") " Oct 30 19:45:59.281680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281579 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home" pod "3c64d263-5c5a-4c5b-891c-4607e6e850a3" (UID: "3c64d263-5c5a-4c5b-891c-4607e6e850a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:59.281680 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281576 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache" pod "3c64d263-5c5a-4c5b-891c-4607e6e850a3" (UID: "3c64d263-5c5a-4c5b-891c-4607e6e850a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:59.281780 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281761 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir" pod "3c64d263-5c5a-4c5b-891c-4607e6e850a3" (UID: "3c64d263-5c5a-4c5b-891c-4607e6e850a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:59.281889 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281871 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:59.281952 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281894 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:59.281952 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.281907 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:59.284296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.284249 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg" pod "3c64d263-5c5a-4c5b-891c-4607e6e850a3" (UID: "3c64d263-5c5a-4c5b-891c-4607e6e850a3"). InnerVolumeSpecName "kube-api-access-t7hxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:59.284296 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.284249 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs" pod "3c64d263-5c5a-4c5b-891c-4607e6e850a3" (UID: "3c64d263-5c5a-4c5b-891c-4607e6e850a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:59.285895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.285869 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm" pod "3c64d263-5c5a-4c5b-891c-4607e6e850a3" (UID: "3c64d263-5c5a-4c5b-891c-4607e6e850a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:59.312126 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.312094 6511 generic.go:356] "Generic (PLEG): container finished" podID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerID="235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3" exitCode=0 Oct 30 19:45:59.312268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.312155 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" Oct 30 19:45:59.312268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.312174 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" event={"ID":"3c64d263-5c5a-4c5b-891c-4607e6e850a3","Type":"ContainerDied","Data":"235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3"} Oct 30 19:45:59.312268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.312228 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz" event={"ID":"3c64d263-5c5a-4c5b-891c-4607e6e850a3","Type":"ContainerDied","Data":"e013278f92e446d98cb54c1b304dac4b2952b634ccd96a2941704f66a15d0481"} Oct 30 19:45:59.312268 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.312253 6511 scope.go:122] "RemoveContainer" containerID="235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3" Oct 30 19:45:59.323588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.323392 6511 scope.go:122] "RemoveContainer" containerID="235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3" Oct 30 19:45:59.323933 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:45:59.323912 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3\": container with ID starting with 235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3 not found: ID does not exist" containerID="235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3" Oct 30 19:45:59.324035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.323943 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3"} err="failed to get container status \"235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3\": rpc error: code = NotFound desc = could not find container \"235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3\": container with ID starting with 235e9fe3e5dbede4f016a6883e657c236c8b9eb1d279342ef8ab254d6bdce4c3 not found: ID does not exist" Oct 30 19:45:59.346222 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.346200 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz"] Oct 30 19:45:59.350029 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.350008 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ce69f3/distinct-2-w3-ce69f3-kserve-699bd94c4c-cvxqz"] Oct 30 19:45:59.383152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.383070 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c64d263-5c5a-4c5b-891c-4607e6e850a3-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:59.383152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.383101 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c64d263-5c5a-4c5b-891c-4607e6e850a3-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:59.383152 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:45:59.383119 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7hxg\" (UniqueName: \"kubernetes.io/projected/3c64d263-5c5a-4c5b-891c-4607e6e850a3-kube-api-access-t7hxg\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:00.096126 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:00.096088 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" path="/var/lib/kubelet/pods/3c64d263-5c5a-4c5b-891c-4607e6e850a3/volumes" Oct 30 19:46:08.168254 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.168215 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" containerID="cri-o://0863844bf6954cb523e0d28dab12270fab477ad261aa4fb2cf123bd95753b36a" gracePeriod=45 Oct 30 19:46:08.192560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.192530 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" containerID="cri-o://9074dec1ef1f9a1d715011bcdc1fed08d04fc2c237fa530d55120fdcf901c862" gracePeriod=45 Oct 30 19:46:08.293058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.293024 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused" Oct 30 19:46:08.373310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.372993 6511 generic.go:356] "Generic (PLEG): container finished" podID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerID="0863844bf6954cb523e0d28dab12270fab477ad261aa4fb2cf123bd95753b36a" exitCode=0 Oct 30 19:46:08.373310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.373101 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" event={"ID":"a258aebe-81aa-4b03-a7ff-0aca873e5cc6","Type":"ContainerDied","Data":"0863844bf6954cb523e0d28dab12270fab477ad261aa4fb2cf123bd95753b36a"} Oct 30 19:46:08.374848 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.374822 6511 generic.go:356] "Generic (PLEG): container finished" podID="a06946fa-d410-46a0-a049-442b7d2f7949" containerID="9074dec1ef1f9a1d715011bcdc1fed08d04fc2c237fa530d55120fdcf901c862" exitCode=0 Oct 30 19:46:08.374977 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.374919 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" event={"ID":"a06946fa-d410-46a0-a049-442b7d2f7949","Type":"ContainerDied","Data":"9074dec1ef1f9a1d715011bcdc1fed08d04fc2c237fa530d55120fdcf901c862"} Oct 30 19:46:08.441982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.441957 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:46:08.470991 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.470961 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm\") pod \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " Oct 30 19:46:08.471146 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.471028 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds\" (UniqueName: \"kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds\") pod \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " Oct 30 19:46:08.471146 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.471105 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir\") pod \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " Oct 30 19:46:08.471269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.471149 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs\") pod \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " Oct 30 19:46:08.473187 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.472029 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir" pod "a258aebe-81aa-4b03-a7ff-0aca873e5cc6" (UID: "a258aebe-81aa-4b03-a7ff-0aca873e5cc6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.474566 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.474540 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds" pod "a258aebe-81aa-4b03-a7ff-0aca873e5cc6" (UID: "a258aebe-81aa-4b03-a7ff-0aca873e5cc6"). InnerVolumeSpecName "kube-api-access-pz7ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:08.475238 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.475197 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs" pod "a258aebe-81aa-4b03-a7ff-0aca873e5cc6" (UID: "a258aebe-81aa-4b03-a7ff-0aca873e5cc6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:08.475534 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.475508 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache\") pod \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " Oct 30 19:46:08.475639 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.475588 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home\") pod \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\" (UID: \"a258aebe-81aa-4b03-a7ff-0aca873e5cc6\") " Oct 30 19:46:08.475888 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.475773 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache" pod "a258aebe-81aa-4b03-a7ff-0aca873e5cc6" (UID: "a258aebe-81aa-4b03-a7ff-0aca873e5cc6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.475978 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.475958 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home" pod "a258aebe-81aa-4b03-a7ff-0aca873e5cc6" (UID: "a258aebe-81aa-4b03-a7ff-0aca873e5cc6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.476106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476083 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.476225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476109 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.476225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476123 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.476225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476138 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.476225 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476152 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7ds\" (UniqueName: \"kubernetes.io/projected/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-kube-api-access-pz7ds\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.476448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476241 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm" pod "a258aebe-81aa-4b03-a7ff-0aca873e5cc6" (UID: "a258aebe-81aa-4b03-a7ff-0aca873e5cc6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.476714 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.476695 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:46:08.537799 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.537768 6511 scope.go:122] "RemoveContainer" containerID="0863844bf6954cb523e0d28dab12270fab477ad261aa4fb2cf123bd95753b36a" Oct 30 19:46:08.553647 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.553619 6511 scope.go:122] "RemoveContainer" containerID="9074dec1ef1f9a1d715011bcdc1fed08d04fc2c237fa530d55120fdcf901c862" Oct 30 19:46:08.577052 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577019 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home\") pod \"a06946fa-d410-46a0-a049-442b7d2f7949\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " Oct 30 19:46:08.577167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577096 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4\" (UniqueName: \"kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4\") pod \"a06946fa-d410-46a0-a049-442b7d2f7949\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " Oct 30 19:46:08.577167 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577142 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir\") pod \"a06946fa-d410-46a0-a049-442b7d2f7949\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " Oct 30 19:46:08.577260 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577215 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache\") pod \"a06946fa-d410-46a0-a049-442b7d2f7949\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " Oct 30 19:46:08.577260 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577244 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm\") pod \"a06946fa-d410-46a0-a049-442b7d2f7949\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " Oct 30 19:46:08.577369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577285 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs\" (UniqueName: \"kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs\") pod \"a06946fa-d410-46a0-a049-442b7d2f7949\" (UID: \"a06946fa-d410-46a0-a049-442b7d2f7949\") " Oct 30 19:46:08.577369 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577305 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home" pod "a06946fa-d410-46a0-a049-442b7d2f7949" (UID: "a06946fa-d410-46a0-a049-442b7d2f7949"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.577502 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577483 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir" pod "a06946fa-d410-46a0-a049-442b7d2f7949" (UID: "a06946fa-d410-46a0-a049-442b7d2f7949"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.577644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577625 6511 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-home\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.577720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577648 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a258aebe-81aa-4b03-a7ff-0aca873e5cc6-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.577720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577662 6511 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-tmp-dir\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.577821 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.577727 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache" pod "a06946fa-d410-46a0-a049-442b7d2f7949" (UID: "a06946fa-d410-46a0-a049-442b7d2f7949"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.579548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.579492 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4" pod "a06946fa-d410-46a0-a049-442b7d2f7949" (UID: "a06946fa-d410-46a0-a049-442b7d2f7949"). InnerVolumeSpecName "kube-api-access-p4fz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:08.579548 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.579537 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm" pod "a06946fa-d410-46a0-a049-442b7d2f7949" (UID: "a06946fa-d410-46a0-a049-442b7d2f7949"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:08.579767 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.579740 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs" pod "a06946fa-d410-46a0-a049-442b7d2f7949" (UID: "a06946fa-d410-46a0-a049-442b7d2f7949"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:08.678320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.678234 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fz4\" (UniqueName: \"kubernetes.io/projected/a06946fa-d410-46a0-a049-442b7d2f7949-kube-api-access-p4fz4\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.678320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.678265 6511 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-model-cache\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.678320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.678276 6511 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a06946fa-d410-46a0-a049-442b7d2f7949-dshm\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:08.678320 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:08.678285 6511 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a06946fa-d410-46a0-a049-442b7d2f7949-tls-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:09.379186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.379143 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" event={"ID":"a258aebe-81aa-4b03-a7ff-0aca873e5cc6","Type":"ContainerDied","Data":"66bad68f97e83e46f25cf306d6c73f06231bdaafe7608e79110c18711efdefc0"} Oct 30 19:46:09.379186 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.379178 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227" Oct 30 19:46:09.379681 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.379199 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" event={"ID":"a06946fa-d410-46a0-a049-442b7d2f7949","Type":"ContainerDied","Data":"36ed3c641f4bb1b85b994418ccc2d7abee3ac7afe76b095eb2ebc09c534e74b7"} Oct 30 19:46:09.379681 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.379182 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd" Oct 30 19:46:09.445064 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.444675 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd"] Oct 30 19:46:09.446365 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.446340 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-w4-ad96a4-kserve-56bbf4dc46-jlxxd"] Oct 30 19:46:09.457819 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.457795 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227"] Oct 30 19:46:09.460310 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:09.460292 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad96a4/distinct-2-w4-ad96a4-kserve-5f774bf95c-dr227"] Oct 30 19:46:10.097370 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:10.097340 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" path="/var/lib/kubelet/pods/a06946fa-d410-46a0-a049-442b7d2f7949/volumes" Oct 30 19:46:10.097732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:10.097718 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" path="/var/lib/kubelet/pods/a258aebe-81aa-4b03-a7ff-0aca873e5cc6/volumes" Oct 30 19:46:10.760473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:10.758385 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj"] Oct 30 19:46:10.760473 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:10.758665 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" containerID="cri-o://c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508" gracePeriod=30 Oct 30 19:46:11.914652 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:11.914614 6511 patch_prober.go:29] interesting pod/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:46:11.915044 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:11.914673 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:46:16.023732 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.023701 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:46:16.055820 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.055788 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.055864 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056009 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.055976 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056013 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056045 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056088 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056119 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056148 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056206 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token\") pod \"f510c1f8-7a68-4839-81d7-f5e952135d2d\" (UID: \"f510c1f8-7a68-4839-81d7-f5e952135d2d\") " Oct 30 19:46:16.056341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056221 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.056341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056305 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:16.056733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056583 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.056733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056589 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.056733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056614 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f510c1f8-7a68-4839-81d7-f5e952135d2d-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.056895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.056737 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.057040 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.057001 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.058402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.058377 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "kube-api-access-qmxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.058763 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.058743 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:16.058949 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.058922 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:16.059327 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.059307 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo" pod "f510c1f8-7a68-4839-81d7-f5e952135d2d" (UID: "f510c1f8-7a68-4839-81d7-f5e952135d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157620 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157654 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157669 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157683 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157696 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f510c1f8-7a68-4839-81d7-f5e952135d2d-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157709 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.157815 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.157723 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxfp\" (UniqueName: \"kubernetes.io/projected/f510c1f8-7a68-4839-81d7-f5e952135d2d-kube-api-access-qmxfp\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:16.418898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.418816 6511 generic.go:356] "Generic (PLEG): container finished" podID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerID="c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508" exitCode=0 Oct 30 19:46:16.418898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.418868 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" Oct 30 19:46:16.419083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.418905 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" event={"ID":"f510c1f8-7a68-4839-81d7-f5e952135d2d","Type":"ContainerDied","Data":"c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508"} Oct 30 19:46:16.419083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.418968 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj" event={"ID":"f510c1f8-7a68-4839-81d7-f5e952135d2d","Type":"ContainerDied","Data":"7fddf5fb46de5012372e556e0e5dbac81156f1cf362d26548a187034bd91c8b4"} Oct 30 19:46:16.419083 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.418996 6511 scope.go:122] "RemoveContainer" containerID="c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508" Oct 30 19:46:16.428058 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.428025 6511 scope.go:122] "RemoveContainer" containerID="c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508" Oct 30 19:46:16.428364 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:46:16.428346 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508\": container with ID starting with c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508 not found: ID does not exist" containerID="c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508" Oct 30 19:46:16.428402 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.428373 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508"} err="failed to get container status \"c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508\": rpc error: code = NotFound desc = could not find container \"c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508\": container with ID starting with c5a42cbca75ac1d65cfef62757dc7640af3f7f521f617f0a81944da01b11f508 not found: ID does not exist" Oct 30 19:46:16.443943 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.443920 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj"] Oct 30 19:46:16.453733 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.453708 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ce69f3-openshift-default-6f98547647-dxnzj"] Oct 30 19:46:16.554330 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.554300 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85"] Oct 30 19:46:16.554560 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:16.554534 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" containerID="cri-o://755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a" gracePeriod=30 Oct 30 19:46:18.094933 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:18.094901 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" path="/var/lib/kubelet/pods/f510c1f8-7a68-4839-81d7-f5e952135d2d/volumes" Oct 30 19:46:21.416999 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.416958 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2"] Oct 30 19:46:21.418192 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.417883 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" containerID="cri-o://214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2" gracePeriod=30 Oct 30 19:46:21.867349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.867321 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:46:21.916922 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.916887 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917107 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.916959 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917107 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.916991 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917335 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917309 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917358 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917379 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:21.917503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917427 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917455 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917503 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917487 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917538 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs\") pod \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\" (UID: \"480d3b55-ff1e-4e1c-a449-a9eea535eb34\") " Oct 30 19:46:21.917791 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917680 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:21.917945 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917889 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:21.917945 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.917912 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:21.918368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.918342 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:21.918500 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.918368 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:21.919252 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.919224 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:21.921159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.920312 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "kube-api-access-cdj8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:21.921159 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.920479 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:21.921846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.921800 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:21.923054 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:21.923027 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token" pod "480d3b55-ff1e-4e1c-a449-a9eea535eb34" (UID: "480d3b55-ff1e-4e1c-a449-a9eea535eb34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019205 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019234 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019244 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019254 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019263 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480d3b55-ff1e-4e1c-a449-a9eea535eb34-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019272 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdj8d\" (UniqueName: \"kubernetes.io/projected/480d3b55-ff1e-4e1c-a449-a9eea535eb34-kube-api-access-cdj8d\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.019279 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.019281 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480d3b55-ff1e-4e1c-a449-a9eea535eb34-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:22.459614 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.459581 6511 generic.go:356] "Generic (PLEG): container finished" podID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerID="755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a" exitCode=0 Oct 30 19:46:22.460012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.459629 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" Oct 30 19:46:22.460012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.459643 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" event={"ID":"480d3b55-ff1e-4e1c-a449-a9eea535eb34","Type":"ContainerDied","Data":"755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a"} Oct 30 19:46:22.460012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.459690 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85" event={"ID":"480d3b55-ff1e-4e1c-a449-a9eea535eb34","Type":"ContainerDied","Data":"022d19e7feeb4c5fe5feb34bcd8e79fb51c3047dea174b88ae731e74cccc9de8"} Oct 30 19:46:22.460012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.459712 6511 scope.go:122] "RemoveContainer" containerID="755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a" Oct 30 19:46:22.469355 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.469337 6511 scope.go:122] "RemoveContainer" containerID="755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a" Oct 30 19:46:22.472933 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:46:22.472908 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a\": container with ID starting with 755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a not found: ID does not exist" containerID="755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a" Oct 30 19:46:22.473029 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.472943 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a"} err="failed to get container status \"755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a\": rpc error: code = NotFound desc = could not find container \"755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a\": container with ID starting with 755bf60691812a8095de1d0467a8e1ddcd728731c737a114d95f4ca50033aa3a not found: ID does not exist" Oct 30 19:46:22.485706 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.485680 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85"] Oct 30 19:46:22.493208 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:22.493186 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-frx85"] Oct 30 19:46:24.095845 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:24.095804 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" path="/var/lib/kubelet/pods/480d3b55-ff1e-4e1c-a449-a9eea535eb34/volumes" Oct 30 19:46:24.377252 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:46:24.377173 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:46:26.708970 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.708947 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768059 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768131 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768188 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768219 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768270 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768308 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768354 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768388 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768488 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token\") pod \"38d4dd33-de2d-44d3-8676-1a6a940d100f\" (UID: \"38d4dd33-de2d-44d3-8676-1a6a940d100f\") " Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768645 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.769036 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.768813 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.769973 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.769891 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.769973 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.769947 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.770341 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.770304 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:26.770796 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.770764 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.772830 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.772802 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:26.775691 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.775654 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "kube-api-access-6db6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:26.789852 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.789816 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:26.790557 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.790529 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy" pod "38d4dd33-de2d-44d3-8676-1a6a940d100f" (UID: "38d4dd33-de2d-44d3-8676-1a6a940d100f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:26.870316 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870278 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870321 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6db6r\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-kube-api-access-6db6r\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870341 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870356 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870371 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870385 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d4dd33-de2d-44d3-8676-1a6a940d100f-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870400 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d4dd33-de2d-44d3-8676-1a6a940d100f-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:26.870518 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:26.870459 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d4dd33-de2d-44d3-8676-1a6a940d100f-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:27.493797 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.493698 6511 generic.go:356] "Generic (PLEG): container finished" podID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerID="214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2" exitCode=0 Oct 30 19:46:27.493797 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.493767 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" event={"ID":"38d4dd33-de2d-44d3-8676-1a6a940d100f","Type":"ContainerDied","Data":"214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2"} Oct 30 19:46:27.494012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.493822 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" event={"ID":"38d4dd33-de2d-44d3-8676-1a6a940d100f","Type":"ContainerDied","Data":"159b5cf1d33ac9b36354cb0491e7a1a2af55e356d8a9b3849256a7b16483ed22"} Oct 30 19:46:27.494012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.493856 6511 scope.go:122] "RemoveContainer" containerID="214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2" Oct 30 19:46:27.494104 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.494013 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2" Oct 30 19:46:27.505653 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.505634 6511 scope.go:122] "RemoveContainer" containerID="214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2" Oct 30 19:46:27.505938 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:46:27.505918 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2\": container with ID starting with 214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2 not found: ID does not exist" containerID="214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2" Oct 30 19:46:27.505990 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.505948 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2"} err="failed to get container status \"214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2\": rpc error: code = NotFound desc = could not find container \"214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2\": container with ID starting with 214e2ac0a718ecd19e2a9a060296390c2e67e849c07740b7b4a13a9aad5d31b2 not found: ID does not exist" Oct 30 19:46:27.529979 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.529896 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2"] Oct 30 19:46:27.542293 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:27.542269 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad96a4-openshift-default-5b7bbc6b7-wkhn2"] Oct 30 19:46:28.094959 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:28.094928 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" path="/var/lib/kubelet/pods/38d4dd33-de2d-44d3-8676-1a6a940d100f/volumes" Oct 30 19:46:46.633193 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633152 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct"] Oct 30 19:46:46.633750 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633727 6511 cpu_manager.go:414] "containerMap: removing container" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633753 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633766 6511 cpu_manager.go:414] "containerMap: removing container" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633774 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633795 6511 cpu_manager.go:414] "containerMap: removing container" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633803 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633829 6511 cpu_manager.go:414] "containerMap: removing container" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" Oct 30 19:46:46.633834 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633836 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633859 6511 cpu_manager.go:414] "containerMap: removing container" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633866 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633883 6511 cpu_manager.go:414] "containerMap: removing container" podUID="9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" containerName="revoke-keys" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633891 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" containerName="revoke-keys" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633908 6511 cpu_manager.go:414] "containerMap: removing container" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633917 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633929 6511 cpu_manager.go:414] "containerMap: removing container" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.633935 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634028 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a258aebe-81aa-4b03-a7ff-0aca873e5cc6" containerName="main" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634039 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d4dd33-de2d-44d3-8676-1a6a940d100f" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634047 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0ffd8f-5549-49ed-9d8c-b30cc8474977" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634054 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="972c6c02-f074-4a31-8cf9-aa5029cc68b4" containerName="cleanup" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634059 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6c6b51-c969-451b-b1fe-6287758d2679" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634066 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" containerName="revoke-keys" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634072 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="480d3b55-ff1e-4e1c-a449-a9eea535eb34" containerName="istio-proxy" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634078 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c64d263-5c5a-4c5b-891c-4607e6e850a3" containerName="main" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634083 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06946fa-d410-46a0-a049-442b7d2f7949" containerName="main" Oct 30 19:46:46.634352 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.634088 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="f510c1f8-7a68-4839-81d7-f5e952135d2d" containerName="istio-proxy" Oct 30 19:46:46.636529 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.636508 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.657011 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.656962 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct"] Oct 30 19:46:46.755299 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755263 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755512 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755332 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755512 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755461 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755512 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755498 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755549 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgbsb\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755620 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755599 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755686 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755630 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755768 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755749 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.755808 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.755790 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856581 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856616 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856625 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856646 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856697 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856717 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856734 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856753 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgbsb\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856774 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.856895 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.856802 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.857263 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.857209 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.857263 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.857242 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.857379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.857318 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.857379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.857337 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.857551 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.857527 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.859387 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.859359 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.859537 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.859518 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.865175 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.865153 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.865363 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.865340 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgbsb\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ndqct\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:46.951240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:46.951187 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:47.246737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.246630 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct"] Oct 30 19:46:47.249900 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:46:47.249858 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f3af5f_c631_454b_a95d_7b0abb0f82de.slice/crio-7dfcd0db8f2bfbee141789713ccd6eb9c25130feec788533669d482f50c21635 WatchSource:0}: Error finding container 7dfcd0db8f2bfbee141789713ccd6eb9c25130feec788533669d482f50c21635: Status 404 returned error can't find the container with id 7dfcd0db8f2bfbee141789713ccd6eb9c25130feec788533669d482f50c21635 Oct 30 19:46:47.252046 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.251973 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:46:47.252134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.252052 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:46:47.252134 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.252099 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:46:47.591617 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.591571 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" event={"ID":"05f3af5f-c631-454b-a95d-7b0abb0f82de","Type":"ContainerStarted","Data":"5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac"} Oct 30 19:46:47.591788 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.591634 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" event={"ID":"05f3af5f-c631-454b-a95d-7b0abb0f82de","Type":"ContainerStarted","Data":"7dfcd0db8f2bfbee141789713ccd6eb9c25130feec788533669d482f50c21635"} Oct 30 19:46:47.622088 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.622028 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podStartSLOduration=1.622011753 podStartE2EDuration="1.622011753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:47.620279745 +0000 UTC m=+1781.643883646" watchObservedRunningTime="2026-10-30 19:46:47.622011753 +0000 UTC m=+1781.645615651" Oct 30 19:46:47.952203 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:47.952116 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:48.953040 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:48.953002 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:46:48.953588 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:48.953055 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 19:46:49.952281 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:49.952247 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:46:49.952469 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:49.952297 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 19:46:50.951724 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:50.951686 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:46:50.952142 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:50.951742 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 19:46:51.951952 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:51.951913 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:46:51.952375 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:51.951977 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 19:46:52.952440 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:52.952388 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:46:52.952925 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:52.952461 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 19:46:53.951631 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:53.951594 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ndqct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:46:53.951818 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:53.951658 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 19:46:54.377269 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:46:54.377230 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:46:55.603378 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:55.603337 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:55.604069 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:55.604047 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:46:55.604792 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:46:55.604768 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:47:06.561379 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:06.561347 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:47:06.577505 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:06.577485 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:47:11.091496 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.091450 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s"] Oct 30 19:47:11.092001 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.091695 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" podUID="9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" containerName="discovery" containerID="cri-o://e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b" gracePeriod=30 Oct 30 19:47:11.336878 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.336854 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:47:11.391898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.391829 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.391898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.391876 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.391898 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.391896 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.392151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.391943 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.392151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.391963 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.392151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.392036 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.392151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.392075 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig\") pod \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\" (UID: \"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a\") " Oct 30 19:47:11.392782 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.392732 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:11.394658 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.394540 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:11.394777 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.394688 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:11.394841 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.394821 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:11.394912 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.394888 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:11.395012 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.394993 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:11.395086 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.395070 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh" pod "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" (UID: "9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a"). InnerVolumeSpecName "kube-api-access-bcsvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:11.493051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493024 6511 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-ca-configmap\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.493051 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493051 6511 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-kubeconfig\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.493240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493061 6511 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-local-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.493240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493071 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.493240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493079 6511 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-istio-csr-dns-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.493240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493088 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcsvh\" (UniqueName: \"kubernetes.io/projected/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-kube-api-access-bcsvh\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.493240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.493098 6511 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a-cacerts\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:11.718150 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.718074 6511 generic.go:356] "Generic (PLEG): container finished" podID="9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" containerID="e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b" exitCode=0 Oct 30 19:47:11.718150 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.718123 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" Oct 30 19:47:11.718329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.718124 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" event={"ID":"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a","Type":"ContainerDied","Data":"e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b"} Oct 30 19:47:11.718329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.718235 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s" event={"ID":"9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a","Type":"ContainerDied","Data":"ae54203d29ebf434a6ecfa3984628dc4675013146226d1d6bf864d02f5285aaf"} Oct 30 19:47:11.718329 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.718260 6511 scope.go:122] "RemoveContainer" containerID="e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b" Oct 30 19:47:11.728319 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.728298 6511 scope.go:122] "RemoveContainer" containerID="e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b" Oct 30 19:47:11.728594 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:47:11.728575 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b\": container with ID starting with e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b not found: ID does not exist" containerID="e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b" Oct 30 19:47:11.728671 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.728602 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b"} err="failed to get container status \"e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b\": rpc error: code = NotFound desc = could not find container \"e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b\": container with ID starting with e2ecd3b18229f9defa514ebfcc62703bbc138a670f54d27e9b3a36f529a8237b not found: ID does not exist" Oct 30 19:47:11.740761 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.740734 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s"] Oct 30 19:47:11.743822 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:11.743790 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p5c6s"] Oct 30 19:47:12.095261 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:47:12.095224 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" path="/var/lib/kubelet/pods/9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a/volumes" Oct 30 19:47:24.376854 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:47:24.376769 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:47:54.408990 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:47:54.408942 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:48:24.412818 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:48:24.412773 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:48:54.377061 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:48:54.376980 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:49:24.376912 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:49:24.376876 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:49:54.380013 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:49:54.379976 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:50:24.409892 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:50:24.409812 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:50:53.031637 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:50:53.031583 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p"] Oct 30 19:50:53.037923 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:50:53.037886 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27026cb4-wbx2p"] Oct 30 19:50:54.096151 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:50:54.096114 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb4cfcf-8979-4b82-b7ea-a92f7de2613b" path="/var/lib/kubelet/pods/9fb4cfcf-8979-4b82-b7ea-a92f7de2613b/volumes" Oct 30 19:50:54.376998 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:50:54.376912 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:51:15.186713 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:51:15.186675 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:51:15.191977 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:51:15.187234 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:51:24.375977 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:51:24.375941 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:51:54.375705 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:51:54.375626 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:52:06.595801 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:52:06.595771 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:52:06.611982 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:52:06.611958 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:52:08.680641 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:52:08.680603 6511 scope.go:122] "RemoveContainer" containerID="18e842cd629393f2160991a977cc411e7b2877b4a569856fd82f1d919a71f3a3" Oct 30 19:52:24.377048 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:52:24.377009 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:52:54.376735 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:52:54.376687 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:53:24.376518 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:53:24.376430 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:53:54.376718 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:53:54.376677 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:54:24.376614 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:54:24.376573 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:54:54.376462 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:54:54.376362 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:55:24.376475 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:55:24.376432 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:55:54.376756 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:55:54.376719 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:56:24.376588 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:56:24.376507 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:56:47.291461 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:47.291407 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct"] Oct 30 19:56:47.291980 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:47.291645 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" containerID="cri-o://5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac" gracePeriod=30 Oct 30 19:56:47.296119 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:47.296096 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m"] Oct 30 19:56:47.296292 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:47.296259 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" containerID="cri-o://c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3" gracePeriod=30 Oct 30 19:56:52.557331 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.557304 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:56:52.560322 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.560303 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:56:52.591862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.591800 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.591862 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.591845 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.591868 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.591888 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592008 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592057 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592051 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592084 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592122 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592135 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592171 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592210 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592235 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592269 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592266 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592297 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592324 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592393 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy\") pod \"625279b6-91d4-4ccf-805d-5d8e74d20500\" (UID: \"625279b6-91d4-4ccf-805d-5d8e74d20500\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592438 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592451 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592445 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592479 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592531 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket\") pod \"05f3af5f-c631-454b-a95d-7b0abb0f82de\" (UID: \"05f3af5f-c631-454b-a95d-7b0abb0f82de\") " Oct 30 19:56:52.592674 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592534 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:52.593147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592876 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.593147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592897 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.593147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592911 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f3af5f-c631-454b-a95d-7b0abb0f82de-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.593147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592904 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.593147 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.592925 6511 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.593679 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.593169 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.593679 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.593498 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:52.593882 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.593860 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.595868 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.595772 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "kube-api-access-dvp4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.596109 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.596086 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.596465 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.596400 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.596881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.596675 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.596881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.596724 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:52.596881 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.596749 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "kube-api-access-bgbsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.597155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.597117 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:52.597358 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.597331 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:52.598371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.598345 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token" pod "625279b6-91d4-4ccf-805d-5d8e74d20500" (UID: "625279b6-91d4-4ccf-805d-5d8e74d20500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.598896 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.598861 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token" pod "05f3af5f-c631-454b-a95d-7b0abb0f82de" (UID: "05f3af5f-c631-454b-a95d-7b0abb0f82de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:52.694259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694222 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694262 6511 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-data\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694278 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694293 6511 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625279b6-91d4-4ccf-805d-5d8e74d20500-istiod-ca-cert\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694308 6511 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-podinfo\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694323 6511 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-workload-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694337 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694350 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694363 6511 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-envoy\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694374 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694384 6511 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f3af5f-c631-454b-a95d-7b0abb0f82de-credential-socket\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694393 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgbsb\" (UniqueName: \"kubernetes.io/projected/05f3af5f-c631-454b-a95d-7b0abb0f82de-kube-api-access-bgbsb\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694402 6511 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-istio-token\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.694586 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.694426 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvp4s\" (UniqueName: \"kubernetes.io/projected/625279b6-91d4-4ccf-805d-5d8e74d20500-kube-api-access-dvp4s\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:52.708481 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.708445 6511 generic.go:356] "Generic (PLEG): container finished" podID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerID="5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac" exitCode=0 Oct 30 19:56:52.708625 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.708553 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" Oct 30 19:56:52.708700 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.708542 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" event={"ID":"05f3af5f-c631-454b-a95d-7b0abb0f82de","Type":"ContainerDied","Data":"5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac"} Oct 30 19:56:52.708762 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.708705 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct" event={"ID":"05f3af5f-c631-454b-a95d-7b0abb0f82de","Type":"ContainerDied","Data":"7dfcd0db8f2bfbee141789713ccd6eb9c25130feec788533669d482f50c21635"} Oct 30 19:56:52.708762 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.708746 6511 scope.go:122] "RemoveContainer" containerID="5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac" Oct 30 19:56:52.710135 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.710115 6511 generic.go:356] "Generic (PLEG): container finished" podID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerID="c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3" exitCode=0 Oct 30 19:56:52.710232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.710145 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" event={"ID":"625279b6-91d4-4ccf-805d-5d8e74d20500","Type":"ContainerDied","Data":"c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3"} Oct 30 19:56:52.710232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.710169 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" Oct 30 19:56:52.710232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.710199 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m" event={"ID":"625279b6-91d4-4ccf-805d-5d8e74d20500","Type":"ContainerDied","Data":"7eddddbc663ea8db620d26c378bd82223efd682f350f4b8cdadd289cca6857e9"} Oct 30 19:56:52.719155 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.719099 6511 scope.go:122] "RemoveContainer" containerID="5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac" Oct 30 19:56:52.719466 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:56:52.719436 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac\": container with ID starting with 5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac not found: ID does not exist" containerID="5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac" Oct 30 19:56:52.719569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.719468 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac"} err="failed to get container status \"5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac\": rpc error: code = NotFound desc = could not find container \"5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac\": container with ID starting with 5dcd377a69ffbc980fcb9ae0241215950e4419ce6a3edfe369f3b7dea77526ac not found: ID does not exist" Oct 30 19:56:52.719569 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.719486 6511 scope.go:122] "RemoveContainer" containerID="c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3" Oct 30 19:56:52.728131 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.728111 6511 scope.go:122] "RemoveContainer" containerID="c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3" Oct 30 19:56:52.728399 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:56:52.728383 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3\": container with ID starting with c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3 not found: ID does not exist" containerID="c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3" Oct 30 19:56:52.728491 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.728406 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3"} err="failed to get container status \"c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3\": rpc error: code = NotFound desc = could not find container \"c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3\": container with ID starting with c5256001030b117937ce630a0f9ddca8ac644da69dcc1d2ea695ec8bed07b8d3 not found: ID does not exist" Oct 30 19:56:52.741360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.741341 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m"] Oct 30 19:56:52.746941 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.746921 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-zl27m"] Oct 30 19:56:52.758184 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.758165 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct"] Oct 30 19:56:52.766921 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:52.766901 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ndqct"] Oct 30 19:56:54.094995 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:54.094964 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" path="/var/lib/kubelet/pods/05f3af5f-c631-454b-a95d-7b0abb0f82de/volumes" Oct 30 19:56:54.095385 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:56:54.095373 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" path="/var/lib/kubelet/pods/625279b6-91d4-4ccf-805d-5d8e74d20500/volumes" Oct 30 19:56:54.377205 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:56:54.377127 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:57:06.631240 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:57:06.631208 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:57:06.646812 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:57:06.646791 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 19:57:24.377382 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:57:24.377339 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:57:54.377627 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:57:54.377541 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:58:24.377208 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:58:24.377165 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:58:47.543696 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.543656 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9"] Oct 30 19:58:47.544106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544053 6511 cpu_manager.go:414] "containerMap: removing container" podUID="9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" containerName="discovery" Oct 30 19:58:47.544106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544069 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" containerName="discovery" Oct 30 19:58:47.544106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544090 6511 cpu_manager.go:414] "containerMap: removing container" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" Oct 30 19:58:47.544106 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544098 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" Oct 30 19:58:47.544259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544108 6511 cpu_manager.go:414] "containerMap: removing container" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" Oct 30 19:58:47.544259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544117 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" Oct 30 19:58:47.544259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544211 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c16b2c4-cf7e-4d70-9ee8-f010bd60eb0a" containerName="discovery" Oct 30 19:58:47.544259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544224 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f3af5f-c631-454b-a95d-7b0abb0f82de" containerName="istio-proxy" Oct 30 19:58:47.544259 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.544232 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="625279b6-91d4-4ccf-805d-5d8e74d20500" containerName="istio-proxy" Oct 30 19:58:47.547696 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.547670 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.561502 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.561479 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9"] Oct 30 19:58:47.575932 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.575896 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.575932 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.575931 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.575951 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4w8\" (UniqueName: \"kubernetes.io/projected/c5516ba3-579a-488d-a197-2d0377d7a84c-kube-api-access-xn4w8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.576006 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576122 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.576037 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5516ba3-579a-488d-a197-2d0377d7a84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.576149 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.576181 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576232 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.576205 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.576333 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.576278 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.677647 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677611 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.677647 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677651 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.677861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677671 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4w8\" (UniqueName: \"kubernetes.io/projected/c5516ba3-579a-488d-a197-2d0377d7a84c-kube-api-access-xn4w8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.677861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677786 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.677861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677815 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5516ba3-579a-488d-a197-2d0377d7a84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677882 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677912 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677943 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678014 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.677982 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.678060 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678250 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.678230 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678368 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.678344 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678448 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.678365 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.678707 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.678682 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5516ba3-579a-488d-a197-2d0377d7a84c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.680154 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.680131 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.680709 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.680686 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.687168 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.687126 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4w8\" (UniqueName: \"kubernetes.io/projected/c5516ba3-579a-488d-a197-2d0377d7a84c-kube-api-access-xn4w8\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.687599 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.687576 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5516ba3-579a-488d-a197-2d0377d7a84c-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-s7gc9\" (UID: \"c5516ba3-579a-488d-a197-2d0377d7a84c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:47.870531 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:47.870495 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:48.150447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.150401 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9"] Oct 30 19:58:48.152402 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:58:48.152374 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5516ba3_579a_488d_a197_2d0377d7a84c.slice/crio-662992e097240672563e4b58f7410710f0dbfb2124df671a3e3b7baa030ba5e0 WatchSource:0}: Error finding container 662992e097240672563e4b58f7410710f0dbfb2124df671a3e3b7baa030ba5e0: Status 404 returned error can't find the container with id 662992e097240672563e4b58f7410710f0dbfb2124df671a3e3b7baa030ba5e0 Oct 30 19:58:48.154740 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.154692 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:58:48.154833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.154774 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:58:48.154833 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.154804 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:58:48.244996 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.244946 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" event={"ID":"c5516ba3-579a-488d-a197-2d0377d7a84c","Type":"ContainerStarted","Data":"5bac24a5bd41784796193bdb5b8baed428c123dcb146a221ee725fccab4c2e09"} Oct 30 19:58:48.245172 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.245021 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" event={"ID":"c5516ba3-579a-488d-a197-2d0377d7a84c","Type":"ContainerStarted","Data":"662992e097240672563e4b58f7410710f0dbfb2124df671a3e3b7baa030ba5e0"} Oct 30 19:58:48.268532 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.268476 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podStartSLOduration=1.26845841 podStartE2EDuration="1.26845841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:48.26644923 +0000 UTC m=+2502.290053134" watchObservedRunningTime="2026-10-30 19:58:48.26845841 +0000 UTC m=+2502.292062313" Oct 30 19:58:48.870639 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:48.870608 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:49.876218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:49.876188 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:50.254870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:50.254794 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:50.255698 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:58:50.255681 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 19:58:54.376354 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:58:54.376317 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:59:02.546487 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.546454 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf"] Oct 30 19:59:02.549847 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.549831 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.562590 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.562567 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf"] Oct 30 19:59:02.713173 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713128 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713178 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713301 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713360 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713341 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713486 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713572 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713520 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsr9\" (UniqueName: \"kubernetes.io/projected/b5fc48e8-e522-4731-bb8c-04e34847732f-kube-api-access-5fsr9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713587 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc48e8-e522-4731-bb8c-04e34847732f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713644 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713617 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.713737 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.713644 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814218 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814197 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814226 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsr9\" (UniqueName: \"kubernetes.io/projected/b5fc48e8-e522-4731-bb8c-04e34847732f-kube-api-access-5fsr9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814263 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc48e8-e522-4731-bb8c-04e34847732f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814292 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814349 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814328 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814365 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814597 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814393 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814711 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814668 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814771 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814707 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814771 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814738 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814780 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814785 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.814870 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.814805 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.815113 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.815093 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc48e8-e522-4731-bb8c-04e34847732f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.817041 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.817020 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.817157 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.817139 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.823180 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.823160 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsr9\" (UniqueName: \"kubernetes.io/projected/b5fc48e8-e522-4731-bb8c-04e34847732f-kube-api-access-5fsr9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.823338 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.823317 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc48e8-e522-4731-bb8c-04e34847732f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-ctqxf\" (UID: \"b5fc48e8-e522-4731-bb8c-04e34847732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:02.864846 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:02.864824 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:03.140447 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.140399 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf"] Oct 30 19:59:03.141965 ip-10-0-140-127 kubenswrapper[6511]: W1030 19:59:03.141935 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fc48e8_e522_4731_bb8c_04e34847732f.slice/crio-8698184be3e7ba346bce5fad30864d4395caf40c21d0ca9b46058afaab1591db WatchSource:0}: Error finding container 8698184be3e7ba346bce5fad30864d4395caf40c21d0ca9b46058afaab1591db: Status 404 returned error can't find the container with id 8698184be3e7ba346bce5fad30864d4395caf40c21d0ca9b46058afaab1591db Oct 30 19:59:03.144621 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.144585 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:59:03.144720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.144664 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:59:03.144720 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.144711 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 19:59:03.314675 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.314642 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" event={"ID":"b5fc48e8-e522-4731-bb8c-04e34847732f","Type":"ContainerStarted","Data":"5334b90da61d3ba5a4ae1e23c204eb75deb9ebb5685a77e6b0fe7925bd9a5719"} Oct 30 19:59:03.314861 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.314693 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" event={"ID":"b5fc48e8-e522-4731-bb8c-04e34847732f","Type":"ContainerStarted","Data":"8698184be3e7ba346bce5fad30864d4395caf40c21d0ca9b46058afaab1591db"} Oct 30 19:59:03.337853 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.337800 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podStartSLOduration=1.3377840509999999 podStartE2EDuration="1.337784051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:03.336331383 +0000 UTC m=+2517.359935282" watchObservedRunningTime="2026-10-30 19:59:03.337784051 +0000 UTC m=+2517.361387950" Oct 30 19:59:03.865410 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:03.865378 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:04.870371 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:04.870339 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:05.325040 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:05.325011 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:05.326035 ip-10-0-140-127 kubenswrapper[6511]: I1030 19:59:05.326013 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 19:59:24.377291 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:59:24.377210 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 19:59:54.400996 ip-10-0-140-127 kubenswrapper[6511]: E1030 19:59:54.400930 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:00:01.610129 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.610068 6511 generic.go:356] "Generic (PLEG): container finished" podID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerID="5334b90da61d3ba5a4ae1e23c204eb75deb9ebb5685a77e6b0fe7925bd9a5719" exitCode=0 Oct 30 20:00:01.610645 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.610142 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" event={"ID":"b5fc48e8-e522-4731-bb8c-04e34847732f","Type":"ContainerDied","Data":"5334b90da61d3ba5a4ae1e23c204eb75deb9ebb5685a77e6b0fe7925bd9a5719"} Oct 30 20:00:01.610645 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.610581 6511 scope.go:122] "RemoveContainer" containerID="5334b90da61d3ba5a4ae1e23c204eb75deb9ebb5685a77e6b0fe7925bd9a5719" Oct 30 20:00:01.611759 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.611725 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 20:00:01.611875 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.611823 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 20:00:01.611875 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.611862 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 20:00:01.611992 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.611934 6511 generic.go:356] "Generic (PLEG): container finished" podID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerID="5bac24a5bd41784796193bdb5b8baed428c123dcb146a221ee725fccab4c2e09" exitCode=0 Oct 30 20:00:01.611992 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.611969 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" event={"ID":"c5516ba3-579a-488d-a197-2d0377d7a84c","Type":"ContainerDied","Data":"5bac24a5bd41784796193bdb5b8baed428c123dcb146a221ee725fccab4c2e09"} Oct 30 20:00:01.612346 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.612302 6511 scope.go:122] "RemoveContainer" containerID="5bac24a5bd41784796193bdb5b8baed428c123dcb146a221ee725fccab4c2e09" Oct 30 20:00:01.613279 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.613253 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 20:00:01.613406 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.613319 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 20:00:01.613406 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:01.613350 6511 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 20:00:02.668297 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.668256 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" event={"ID":"b5fc48e8-e522-4731-bb8c-04e34847732f","Type":"ContainerStarted","Data":"6e5a025a6093cf8c8ff2fe41910740c4c0fd6725046ab8e3df6ba30f070a62d3"} Oct 30 20:00:02.671070 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.671029 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" event={"ID":"c5516ba3-579a-488d-a197-2d0377d7a84c","Type":"ContainerStarted","Data":"2c119c4baeb97404ae0ceca868b3c2468e512ad2c8d2e1ea9ebbf3c94d11018d"} Oct 30 20:00:02.870442 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.865830 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 20:00:02.870442 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.867677 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:02.870442 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.867735 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:02.873473 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.872763 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 20:00:02.874540 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.874512 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:02.874700 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:02.874561 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:03.873862 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:03.873819 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:03.874852 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:03.874822 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:03.875434 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:03.874762 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:03.875586 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:03.875565 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:04.866881 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:04.866839 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:04.867197 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:04.867163 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:04.870854 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:04.870826 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:04.870979 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:04.870883 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:05.258493 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:05.257511 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 20:00:05.325554 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:05.325519 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 20:00:05.867846 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:05.867811 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:05.868129 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:05.868096 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:05.870955 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:05.870920 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:05.871084 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:05.870976 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:06.865868 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:06.865825 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:06.866507 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:06.866475 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:06.871072 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:06.871034 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:06.871180 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:06.871086 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:07.865493 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:07.865454 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:07.865674 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:07.865521 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:07.871773 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:07.871745 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:07.872096 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:07.871789 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:08.866162 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:08.866120 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 20:00:08.866492 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:08.866191 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 20:00:08.871431 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:08.871379 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:00:08.871569 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:08.871452 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 20:00:10.872391 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:10.872348 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:10.873031 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:10.872998 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:10.906572 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:10.906543 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:10.906753 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:10.906592 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:11.915573 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:11.915533 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:11.916032 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:11.915590 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:11.951440 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:11.950507 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:11.951440 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:11.950570 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:12.916235 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:12.916189 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:12.916691 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:12.916254 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:12.951035 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:12.950996 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:12.951200 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:12.951058 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:13.957660 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:13.957596 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:13.958178 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:13.957709 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:13.992561 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:13.992527 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:13.992715 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:13.992575 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:14.658608 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:14.658573 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 20:00:14.660827 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:14.660805 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" Oct 30 20:00:15.034693 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:15.034599 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:15.034693 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:15.034682 6511 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:15.180746 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:15.180714 6511 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 20:00:15.181739 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:00:15.181718 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" Oct 30 20:00:24.377222 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:00:24.377184 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:00:54.378161 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:00:54.378080 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:01:24.376794 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:01:24.376758 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:01:54.376691 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:01:54.376650 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:02:06.666702 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:06.666670 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 20:02:06.687118 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:06.687093 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 20:02:15.186123 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:15.186080 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-x2wnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:02:15.186859 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:15.186822 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-x2wnb" podUID="ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:15.702601 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:15.702556 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:02:15.702813 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:15.702653 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:24.377023 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:02:24.376937 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:02:30.713549 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:30.713514 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-s7gc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:02:30.713942 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:30.713584 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-s7gc9" podUID="c5516ba3-579a-488d-a197-2d0377d7a84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:31.239439 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:31.235520 6511 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-ctqxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:02:31.239439 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:02:31.235585 6511 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-ctqxf" podUID="b5fc48e8-e522-4731-bb8c-04e34847732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:54.377517 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:02:54.377478 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:03:22.588663 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.588626 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh"] Oct 30 20:03:22.591652 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.591636 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.606578 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.606554 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh"] Oct 30 20:03:22.711123 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.711089 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc70364c-2f24-4f51-aaa8-e87047422cb1-openshift-service-ca\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.711315 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.711170 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc70364c-2f24-4f51-aaa8-e87047422cb1-maas-api-tls\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.711315 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.711255 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnwkl\" (UniqueName: \"kubernetes.io/projected/dc70364c-2f24-4f51-aaa8-e87047422cb1-kube-api-access-wnwkl\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.711315 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.711307 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc70364c-2f24-4f51-aaa8-e87047422cb1-metrics-certs\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.812523 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.812480 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc70364c-2f24-4f51-aaa8-e87047422cb1-openshift-service-ca\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.812523 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.812527 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc70364c-2f24-4f51-aaa8-e87047422cb1-maas-api-tls\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.812797 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.812644 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnwkl\" (UniqueName: \"kubernetes.io/projected/dc70364c-2f24-4f51-aaa8-e87047422cb1-kube-api-access-wnwkl\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.812797 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.812692 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc70364c-2f24-4f51-aaa8-e87047422cb1-metrics-certs\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.813207 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.813175 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc70364c-2f24-4f51-aaa8-e87047422cb1-openshift-service-ca\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.815107 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.815079 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc70364c-2f24-4f51-aaa8-e87047422cb1-maas-api-tls\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.815195 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.815142 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc70364c-2f24-4f51-aaa8-e87047422cb1-metrics-certs\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.824396 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.824367 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnwkl\" (UniqueName: \"kubernetes.io/projected/dc70364c-2f24-4f51-aaa8-e87047422cb1-kube-api-access-wnwkl\") pod \"maas-api-79c7b97c4b-6grfh\" (UID: \"dc70364c-2f24-4f51-aaa8-e87047422cb1\") " pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:22.906341 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:22.906267 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:23.182178 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:23.182150 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh"] Oct 30 20:03:23.183912 ip-10-0-140-127 kubenswrapper[6511]: W1030 20:03:23.183889 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc70364c_2f24_4f51_aaa8_e87047422cb1.slice/crio-5b4afe98e3f182d8ea468730a88733b36ea5fe8bc6d0b1ab8f8fb969ea7d17d5 WatchSource:0}: Error finding container 5b4afe98e3f182d8ea468730a88733b36ea5fe8bc6d0b1ab8f8fb969ea7d17d5: Status 404 returned error can't find the container with id 5b4afe98e3f182d8ea468730a88733b36ea5fe8bc6d0b1ab8f8fb969ea7d17d5 Oct 30 20:03:23.185212 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:23.185193 6511 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:03:23.790706 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:23.790670 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" event={"ID":"dc70364c-2f24-4f51-aaa8-e87047422cb1","Type":"ContainerStarted","Data":"8a62da4e2472408bd1b001a4de1a44bf5d5a283efc817059138cf60668751378"} Oct 30 20:03:23.791185 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:23.790726 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" event={"ID":"dc70364c-2f24-4f51-aaa8-e87047422cb1","Type":"ContainerStarted","Data":"5b4afe98e3f182d8ea468730a88733b36ea5fe8bc6d0b1ab8f8fb969ea7d17d5"} Oct 30 20:03:23.791185 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:23.790810 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:23.815126 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:23.815087 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" podStartSLOduration=1.395573387 podStartE2EDuration="1.815074427s" totalImagesPullingTime="419.50104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:23.813670354 +0000 UTC m=+2777.837274253" watchObservedRunningTime="2026-10-30 20:03:23.815074427 +0000 UTC m=+2777.838678326" Oct 30 20:03:24.377408 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:03:24.377372 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:03:29.800324 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:29.800296 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79c7b97c4b-6grfh" Oct 30 20:03:29.878884 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:29.878848 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl"] Oct 30 20:03:29.879111 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:29.879085 6511 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" podUID="9e804365-db54-41d4-8d00-6c84037fb091" containerName="maas-api" containerID="cri-o://ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb" gracePeriod=30 Oct 30 20:03:31.131944 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.131918 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 20:03:31.294267 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.294180 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4\" (UniqueName: \"kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4\") pod \"9e804365-db54-41d4-8d00-6c84037fb091\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " Oct 30 20:03:31.294470 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.294294 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca\") pod \"9e804365-db54-41d4-8d00-6c84037fb091\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " Oct 30 20:03:31.294470 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.294335 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs\") pod \"9e804365-db54-41d4-8d00-6c84037fb091\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " Oct 30 20:03:31.294470 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.294381 6511 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls\") pod \"9e804365-db54-41d4-8d00-6c84037fb091\" (UID: \"9e804365-db54-41d4-8d00-6c84037fb091\") " Oct 30 20:03:31.294654 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.294620 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca" pod "9e804365-db54-41d4-8d00-6c84037fb091" (UID: "9e804365-db54-41d4-8d00-6c84037fb091"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:31.294753 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.294737 6511 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e804365-db54-41d4-8d00-6c84037fb091-openshift-service-ca\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:31.296594 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.296566 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs" pod "9e804365-db54-41d4-8d00-6c84037fb091" (UID: "9e804365-db54-41d4-8d00-6c84037fb091"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:31.296703 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.296616 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls" pod "9e804365-db54-41d4-8d00-6c84037fb091" (UID: "9e804365-db54-41d4-8d00-6c84037fb091"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:31.296703 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.296663 6511 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4" pod "9e804365-db54-41d4-8d00-6c84037fb091" (UID: "9e804365-db54-41d4-8d00-6c84037fb091"). InnerVolumeSpecName "kube-api-access-2n2d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:31.396090 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.396059 6511 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n2d4\" (UniqueName: \"kubernetes.io/projected/9e804365-db54-41d4-8d00-6c84037fb091-kube-api-access-2n2d4\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:31.396090 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.396087 6511 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-metrics-certs\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:31.396248 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.396102 6511 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e804365-db54-41d4-8d00-6c84037fb091-maas-api-tls\") on node \"ip-10-0-140-127.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:31.827261 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.827222 6511 generic.go:356] "Generic (PLEG): container finished" podID="9e804365-db54-41d4-8d00-6c84037fb091" containerID="ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb" exitCode=0 Oct 30 20:03:31.827459 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.827274 6511 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" Oct 30 20:03:31.827459 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.827304 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" event={"ID":"9e804365-db54-41d4-8d00-6c84037fb091","Type":"ContainerDied","Data":"ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb"} Oct 30 20:03:31.827459 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.827382 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl" event={"ID":"9e804365-db54-41d4-8d00-6c84037fb091","Type":"ContainerDied","Data":"9cedbf45b2e8076a8167c807b509369ae6c6d8bf1045217ec4b3ff66adc926eb"} Oct 30 20:03:31.827459 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.827404 6511 scope.go:122] "RemoveContainer" containerID="ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb" Oct 30 20:03:31.837113 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.837097 6511 scope.go:122] "RemoveContainer" containerID="ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb" Oct 30 20:03:31.837390 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:03:31.837372 6511 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb\": container with ID starting with ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb not found: ID does not exist" containerID="ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb" Oct 30 20:03:31.837461 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.837399 6511 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb"} err="failed to get container status \"ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb\": rpc error: code = NotFound desc = could not find container \"ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb\": container with ID starting with ce3012d05d0c795c6237f160aebcf097f44e1390f28ddfcc7493e2cf37236ecb not found: ID does not exist" Oct 30 20:03:31.851484 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.851459 6511 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl"] Oct 30 20:03:31.863460 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:31.863438 6511 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-585bcbc4bb-npjpl"] Oct 30 20:03:32.094851 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:03:32.094773 6511 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e804365-db54-41d4-8d00-6c84037fb091" path="/var/lib/kubelet/pods/9e804365-db54-41d4-8d00-6c84037fb091/volumes" Oct 30 20:03:54.376821 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:03:54.376739 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:04:24.378171 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:04:24.378131 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:04:54.377392 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:04:54.377355 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:05:24.377325 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:05:24.377249 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:05:54.378043 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:05:54.378005 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:06:24.377464 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:06:24.377406 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:06:54.378338 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:06:54.378259 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:07:00.596601 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:00.596565 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-79xzr_14ed5071-2475-47ad-b73b-83519f55840e/manager/0.log" Oct 30 20:07:00.783777 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:00.783734 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nhvrm_03278d76-60bf-42ad-acd7-0ad099d16707/manager/0.log" Oct 30 20:07:00.979975 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:00.979886 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-c6ct8_1951dba1-e295-4d07-a375-c9ac2992cd5f/manager/0.log" Oct 30 20:07:01.650666 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:01.650632 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s6mpf_67077e2a-7385-4c9c-8f36-ec005a1c2b7d/manager/0.log" Oct 30 20:07:02.654573 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:02.654541 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79c7b97c4b-6grfh_dc70364c-2f24-4f51-aaa8-e87047422cb1/maas-api/0.log" Oct 30 20:07:04.134981 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:04.134950 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-k4w5c_b4483e14-f12b-4836-afb5-e6ca25af6663/postgres/0.log" Oct 30 20:07:06.699540 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:06.699511 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 20:07:06.730425 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:06.730376 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nbd9r_a3ce0b21-0196-4d7c-9b36-99ec47e7d057/ovn-acl-logging/0.log" Oct 30 20:07:08.112142 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:08.112105 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-fsk2s_82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43/kube-auth-proxy/0.log" Oct 30 20:07:08.842476 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:08.842434 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/1.log" Oct 30 20:07:09.814391 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:09.814363 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/1.log" Oct 30 20:07:10.003862 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:10.003830 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/1.log" Oct 30 20:07:11.741823 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:11.741780 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/0.log" Oct 30 20:07:12.482835 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:12.482798 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/0.log" Oct 30 20:07:12.642532 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:12.642485 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/0.log" Oct 30 20:07:24.376471 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:07:24.376428 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:07:24.992430 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:24.992388 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-79xzr_14ed5071-2475-47ad-b73b-83519f55840e/manager/0.log" Oct 30 20:07:25.196348 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:25.196316 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-nhvrm_03278d76-60bf-42ad-acd7-0ad099d16707/manager/0.log" Oct 30 20:07:25.412869 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:25.412840 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-c6ct8_1951dba1-e295-4d07-a375-c9ac2992cd5f/manager/0.log" Oct 30 20:07:26.079609 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:26.079573 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s6mpf_67077e2a-7385-4c9c-8f36-ec005a1c2b7d/manager/0.log" Oct 30 20:07:26.898468 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:26.898436 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79c7b97c4b-6grfh_dc70364c-2f24-4f51-aaa8-e87047422cb1/maas-api/0.log" Oct 30 20:07:28.443672 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:28.443643 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-k4w5c_b4483e14-f12b-4836-afb5-e6ca25af6663/postgres/0.log" Oct 30 20:07:31.877036 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:31.876984 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-fsk2s_82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43/kube-auth-proxy/0.log" Oct 30 20:07:32.653612 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:32.653577 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/1.log" Oct 30 20:07:33.655285 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:33.652996 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/1.log" Oct 30 20:07:33.844803 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:07:33.844772 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/1.log" Oct 30 20:07:54.377148 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:07:54.377110 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:08:24.376898 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:08:24.376813 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:08:54.376752 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:08:54.376711 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:09:07.849285 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:07.849257 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f6wtd_da8143dd-2ae4-4932-91b9-9215cf9cd469/global-pull-secret-syncer/0.log" Oct 30 20:09:08.089495 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:08.089458 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hcv7r_950c0201-b4d4-4b5a-9eb2-c0deb0efe1d4/konnectivity-agent/0.log" Oct 30 20:09:08.234345 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:08.234266 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-127.us-east-2.compute.internal_4860c9cfd0c091391ec52fd8f7245c90/haproxy/0.log" Oct 30 20:09:17.425670 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.425644 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfp6b_ff4585e9-d9f1-4736-aab7-485d22d6888c/node-exporter/0.log" Oct 30 20:09:17.465794 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.465767 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfp6b_ff4585e9-d9f1-4736-aab7-485d22d6888c/kube-rbac-proxy/0.log" Oct 30 20:09:17.501978 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.501950 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfp6b_ff4585e9-d9f1-4736-aab7-485d22d6888c/init-textfile/0.log" Oct 30 20:09:17.641238 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.641202 6511 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g"] Oct 30 20:09:17.641623 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.641608 6511 cpu_manager.go:414] "containerMap: removing container" podUID="9e804365-db54-41d4-8d00-6c84037fb091" containerName="maas-api" Oct 30 20:09:17.641623 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.641624 6511 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e804365-db54-41d4-8d00-6c84037fb091" containerName="maas-api" Oct 30 20:09:17.641732 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.641706 6511 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e804365-db54-41d4-8d00-6c84037fb091" containerName="maas-api" Oct 30 20:09:17.644667 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.644643 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.650406 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.650383 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9glm\"/\"openshift-service-ca.crt\"" Oct 30 20:09:17.650530 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.650517 6511 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x9glm\"/\"kube-root-ca.crt\"" Oct 30 20:09:17.655564 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.655545 6511 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x9glm\"/\"default-dockercfg-fzhm4\"" Oct 30 20:09:17.661829 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.661808 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g"] Oct 30 20:09:17.695461 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.695394 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/prometheus/0.log" Oct 30 20:09:17.700873 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.700842 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-sys\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.701009 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.700880 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-proc\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.701074 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.701002 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgl7d\" (UniqueName: \"kubernetes.io/projected/6bf85c5c-8dff-4696-bd9f-c7268273c856-kube-api-access-rgl7d\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.701123 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.701083 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-lib-modules\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.701164 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.701148 6511 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-podres\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.742777 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.742747 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/config-reloader/0.log" Oct 30 20:09:17.769785 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.769761 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/thanos-sidecar/0.log" Oct 30 20:09:17.802034 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802012 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-lib-modules\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802149 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802092 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-podres\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802215 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802152 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-lib-modules\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802215 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802162 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-sys\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802215 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802191 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-proc\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802215 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802198 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-podres\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802393 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802256 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-sys\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802393 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802285 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6bf85c5c-8dff-4696-bd9f-c7268273c856-proc\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.802393 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.802286 6511 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgl7d\" (UniqueName: \"kubernetes.io/projected/6bf85c5c-8dff-4696-bd9f-c7268273c856-kube-api-access-rgl7d\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.806235 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.806214 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/kube-rbac-proxy-web/0.log" Oct 30 20:09:17.811391 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.811367 6511 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgl7d\" (UniqueName: \"kubernetes.io/projected/6bf85c5c-8dff-4696-bd9f-c7268273c856-kube-api-access-rgl7d\") pod \"perf-node-gather-daemonset-7xn5g\" (UID: \"6bf85c5c-8dff-4696-bd9f-c7268273c856\") " pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.845488 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.845466 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/kube-rbac-proxy/0.log" Oct 30 20:09:17.879883 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.879844 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/kube-rbac-proxy-thanos/0.log" Oct 30 20:09:17.915055 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.915035 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2828c894-def5-44ea-91ba-b3560cd7419f/init-config-reloader/0.log" Oct 30 20:09:17.954327 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.954265 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7mc52_752b9e9a-aba1-4aee-8e54-83aa3ea6c854/prometheus-operator/0.log" Oct 30 20:09:17.956843 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.956827 6511 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:17.989954 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:17.989934 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7mc52_752b9e9a-aba1-4aee-8e54-83aa3ea6c854/kube-rbac-proxy/0.log" Oct 30 20:09:18.204704 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.204638 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7646648d5d-jd8l2_69ea1b29-9274-49ce-a4b1-81a8aa191296/thanos-query/0.log" Oct 30 20:09:18.230607 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.230586 6511 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g"] Oct 30 20:09:18.232744 ip-10-0-140-127 kubenswrapper[6511]: W1030 20:09:18.232711 6511 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6bf85c5c_8dff_4696_bd9f_c7268273c856.slice/crio-5bd6b401bb6eb0959c41edc4242f394db195c1e6640ff548afe6cea106f48172 WatchSource:0}: Error finding container 5bd6b401bb6eb0959c41edc4242f394db195c1e6640ff548afe6cea106f48172: Status 404 returned error can't find the container with id 5bd6b401bb6eb0959c41edc4242f394db195c1e6640ff548afe6cea106f48172 Oct 30 20:09:18.239146 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.239128 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7646648d5d-jd8l2_69ea1b29-9274-49ce-a4b1-81a8aa191296/kube-rbac-proxy-web/0.log" Oct 30 20:09:18.290043 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.290025 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7646648d5d-jd8l2_69ea1b29-9274-49ce-a4b1-81a8aa191296/kube-rbac-proxy/0.log" Oct 30 20:09:18.323677 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.323652 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7646648d5d-jd8l2_69ea1b29-9274-49ce-a4b1-81a8aa191296/prom-label-proxy/0.log" Oct 30 20:09:18.356318 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.356287 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7646648d5d-jd8l2_69ea1b29-9274-49ce-a4b1-81a8aa191296/kube-rbac-proxy-rules/0.log" Oct 30 20:09:18.383204 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.383182 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7646648d5d-jd8l2_69ea1b29-9274-49ce-a4b1-81a8aa191296/kube-rbac-proxy-metrics/0.log" Oct 30 20:09:18.578030 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.577991 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" event={"ID":"6bf85c5c-8dff-4696-bd9f-c7268273c856","Type":"ContainerStarted","Data":"5327d1cdd36bd7004d2211b1c1a45ee56d5d47a52d35ef03a3db54b8eb124d97"} Oct 30 20:09:18.578541 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.578049 6511 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" event={"ID":"6bf85c5c-8dff-4696-bd9f-c7268273c856","Type":"ContainerStarted","Data":"5bd6b401bb6eb0959c41edc4242f394db195c1e6640ff548afe6cea106f48172"} Oct 30 20:09:18.578541 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.578078 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:18.598173 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:18.598129 6511 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" podStartSLOduration=1.598117576 podStartE2EDuration="1.598117576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:18.5955983 +0000 UTC m=+3132.619202198" watchObservedRunningTime="2026-10-30 20:09:18.598117576 +0000 UTC m=+3132.621721475" Oct 30 20:09:22.739467 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:22.739432 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5vdt7_d0eda631-51b9-4c18-ad28-99ebb93e0db6/dns/0.log" Oct 30 20:09:22.763063 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:22.763046 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5vdt7_d0eda631-51b9-4c18-ad28-99ebb93e0db6/kube-rbac-proxy/0.log" Oct 30 20:09:22.899350 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:22.899323 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kxt77_32f31184-b3c4-4a0e-8976-1d91195570e8/dns-node-resolver/0.log" Oct 30 20:09:23.649037 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:23.648955 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4wfl7_a00a389a-2d47-4c51-9e90-85e9ffe3c9d6/node-ca/0.log" Oct 30 20:09:24.377698 ip-10-0-140-127 kubenswrapper[6511]: E1030 20:09:24.377658 6511 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kkdfr" containerName="extractor" Oct 30 20:09:24.592262 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:24.592231 6511 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x9glm/perf-node-gather-daemonset-7xn5g" Oct 30 20:09:27.135157 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.135129 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-fsk2s_82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43/kube-auth-proxy/0.log" Oct 30 20:09:27.287354 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.287317 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/0.log" Oct 30 20:09:27.287645 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.287629 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/1.log" Oct 30 20:09:27.436880 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.436802 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/1.log" Oct 30 20:09:27.436880 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.436839 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/0.log" Oct 30 20:09:27.475875 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.475845 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/0.log" Oct 30 20:09:27.480757 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:27.480727 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/1.log" Oct 30 20:09:29.568296 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:29.568266 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4wmdq_68413742-d011-4c65-8ce8-2cf80d43ac1a/serve-healthcheck-canary/0.log" Oct 30 20:09:29.916561 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:29.916489 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5f4bd96d-fsk2s_82bbe7d5-5b5b-41cc-bb54-2b144b9d7b43/kube-auth-proxy/0.log" Oct 30 20:09:29.989694 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:29.989664 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/1.log" Oct 30 20:09:29.989842 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:29.989827 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-ctqxf_b5fc48e8-e522-4731-bb8c-04e34847732f/istio-proxy/0.log" Oct 30 20:09:30.077169 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.077141 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/0.log" Oct 30 20:09:30.077434 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.077396 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-s7gc9_c5516ba3-579a-488d-a197-2d0377d7a84c/istio-proxy/1.log" Oct 30 20:09:30.091435 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.091353 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/0.log" Oct 30 20:09:30.095468 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.095449 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-x2wnb_ad3a8e75-ee0e-4ba7-8aaa-ad3da51a227f/istio-proxy/1.log" Oct 30 20:09:30.235003 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.234925 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkdfr_69df0eef-ee4f-4135-ab6c-01e806705d4a/kube-rbac-proxy/0.log" Oct 30 20:09:30.272358 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.272323 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkdfr_69df0eef-ee4f-4135-ab6c-01e806705d4a/exporter/0.log" Oct 30 20:09:30.334530 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:30.334501 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkdfr_69df0eef-ee4f-4135-ab6c-01e806705d4a/extractor/0.log" Oct 30 20:09:33.079938 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:33.079909 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-68fb5c5984-qhdzc_46143cf4-b608-4435-9588-d265ebe5f5ab/manager/0.log" Oct 30 20:09:33.109610 ip-10-0-140-127 kubenswrapper[6511]: I1030 20:09:33.109577 6511 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-68fb5c5984-qhdzc_46143cf4-b608-4435-9588-d265ebe5f5ab/copy-manifests/0.log"